Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 1 | # mojo_benchmark |
| 2 | |
| 3 | `mojo_benchmark` allows you to run performance tests for any Mojo application |
| 4 | that participates in the [tracing |
Viet-Trung Luu | 0f4f3ba | 2015-10-10 01:08:40 -0700 | [diff] [blame] | 5 | ecosystem](https://github.com/domokit/mojo/blob/master/mojo/services/tracing/interfaces/tracing.mojom) |
Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 6 | with no app changes required. |
| 7 | |
| 8 | The script reads a list of benchmarks to run from a file, runs each benchmark in |
| 9 | controlled caching conditions with tracing enabled and performs specified |
| 10 | measurements on the collected trace data. |
| 11 | |
| 12 | ## Defining benchmarks |
| 13 | |
| 14 | `mojo_benchmark` runs performance tests defined in a benchmark file. The |
Przemyslaw Pietrzkiewicz | 158137d | 2015-10-28 16:47:01 +0100 | [diff] [blame] | 15 | benchmark file is a Python program setting a dictionary of the following format: |
Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 16 | |
| 17 | ```python |
| 18 | benchmarks = [ |
| 19 | { |
| 20 | 'name': '<name of the benchmark>', |
| 21 | 'app': '<url of the app to benchmark>', |
| 22 | 'shell-args': [], |
| 23 | 'duration': <duration in seconds>, |
| 24 | |
| 25 | # List of measurements to make. |
| 26 | 'measurements': [ |
Przemyslaw Pietrzkiewicz | 158137d | 2015-10-28 16:47:01 +0100 | [diff] [blame] | 27 | { |
| 28 | 'name': <my_measurement>, |
| 29 | 'spec': <spec>, |
| 30 | }, |
| 31 | (...) |
Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 32 | ], |
| 33 | }, |
| 34 | ] |
| 35 | ``` |
| 36 | |
Przemyslaw Pietrzkiewicz | 158137d | 2015-10-28 16:47:01 +0100 | [diff] [blame] | 37 | The benchmark file may reference the `target_os` global that will be any of |
| 38 | ('android', 'linux'), indicating the system on which the benchmarks are run. |
| 39 | |
| 40 | ### Measurement specs |
| 41 | |
Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 42 | The following types of measurements are available: |
| 43 | |
Przemyslaw Pietrzkiewicz | 158137d | 2015-10-28 16:47:01 +0100 | [diff] [blame] | 44 | - `time_until` |
| 45 | - `time_between` |
| 46 | - `avg_duration` |
| 47 | - `percentile_duration` |
| 48 | |
| 49 | `time_until` records the time until the first occurence of the targeted event. |
| 50 | The underlying benchmark runner records the time origin just before issuing the |
| 51 | connection call to the application being benchmarked. Results of `time_until` |
| 52 | measurements are relative to this time. Spec format: |
| 53 | |
| 54 | ``` |
| 55 | 'time_until/<category>/<event>' |
| 56 | ``` |
| 57 | |
| 58 | `time_between` records the time between the first occurence of the first |
| 59 | targeted event and the first occurence of the second targeted event. Spec |
| 60 | format: |
| 61 | |
| 62 | ``` |
| 63 | 'time_between/<category1>/<event1>/<category2>/<event2>' |
| 64 | ``` |
| 65 | |
| 66 | `avg_duration` records the average duration of all occurences of the targeted |
| 67 | event. Spec format: |
| 68 | |
| 69 | ``` |
| 70 | 'avg_duration/<category>/<event>' |
| 71 | ``` |
| 72 | |
| 73 | `percentile_duration` records the value at the given percentile of durations of |
| 74 | all occurences of the targeted event. Spec format: |
| 75 | |
| 76 | ``` |
| 77 | 'percentile_duration/<category>/<event>/<percentile>' |
| 78 | ``` |
| 79 | |
| 80 | where `<percentile>` is a number between 0.0 and 0.1. |
Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 81 | |
| 82 | ## Caching |
| 83 | |
Przemyslaw Pietrzkiewicz | e66c629 | 2015-09-16 18:09:27 +0200 | [diff] [blame] | 84 | The script runs each benchmark twice. The first run (**cold start**) clears |
| 85 | caches of the following apps on startup: |
Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 86 | |
Przemyslaw Pietrzkiewicz | 158137d | 2015-10-28 16:47:01 +0100 | [diff] [blame] | 87 | - `network_service.mojo` |
| 88 | - `url_response_disk_cache.mojo` |
Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 89 | |
| 90 | The second run (**warm start**) runs immediately afterwards, without clearing |
| 91 | any caches. |
| 92 | |
Przemyslaw Pietrzkiewicz | 845d098 | 2015-09-14 15:55:43 +0200 | [diff] [blame] | 93 | ## Example |
| 94 | |
| 95 | For an app that records a trace event named "initialized" in category "my_app" |
| 96 | once its initialization is complete, we can benchmark the initialization time of |
| 97 | the app (from the moment someone tries to connect to it to the app completing |
| 98 | its initialization) using the following benchmark file: |
| 99 | |
| 100 | ```python |
| 101 | benchmarks = [ |
| 102 | { |
| 103 | 'name': 'My app initialization', |
| 104 | 'app': 'https://my_domain/my_app.mojo', |
| 105 | 'duration': 10, |
| 106 | 'measurements': [ |
| 107 | 'time_until/my_app/initialized', |
| 108 | ], |
| 109 | }, |
| 110 | ] |
| 111 | ``` |
Przemyslaw Pietrzkiewicz | 6ead3a9 | 2015-10-28 10:14:37 +0100 | [diff] [blame] | 112 | |
| 113 | ## Dashboard |
| 114 | |
| 115 | `mojo_benchmark` supports uploading the results to an instance of a Catapult |
| 116 | performance dashboard. In order to upload the results of a run to performance |
| 117 | dashboard, pass the `--upload` flag along with required meta-data describing the |
| 118 | data being uploaded: |
| 119 | |
| 120 | ```sh |
| 121 | mojo_benchmark \ |
| 122 | --upload \ |
| 123 | --master-name my-master \ |
| 124 | --bot-name my-bot \ |
| 125 | --test-name my-test-suite |
| 126 | --builder-name my-builder \ |
| 127 | --build-number my-build |
| 128 | --server-url http://my-server.example.com |
| 129 | ``` |
| 130 | |
| 131 | If no `--server-url` is specified, the script assumes that a local instance of |
| 132 | the dashboard is running at `http://localhost:8080`. The script assumes that the |
| 133 | working directory from which it is called is a git repository and queries it to |
| 134 | determine the sequential number identifying the revision (as the number of |
| 135 | commits in the current branch in the repository). |
| 136 | |
| 137 | For more information refer to: |
| 138 | |
| 139 | - [Catapult project](https://github.com/catapult-project/catapult) |