Centralize mojo_benchmark documentation. This patch removes the duplication between mojo_benchmark built-in help and mojo_benchmark.md documentation. The built-in help now points to the Markdown documentation. R=qsr@chromium.org Review URL: https://codereview.chromium.org/1423233002 . Cr-Mirrored-From: https://github.com/domokit/mojo Cr-Mirrored-Commit: 158137d99a91b98774f3b13787b8b9d6fcde4958
diff --git a/docs/mojo_benchmark.md b/docs/mojo_benchmark.md index bcca1ad..f9cb773 100644 --- a/docs/mojo_benchmark.md +++ b/docs/mojo_benchmark.md
@@ -12,7 +12,7 @@ ## Defining benchmarks `mojo_benchmark` runs performance tests defined in a benchmark file. The -benchmark file is a Python dictionary of the following format: +benchmark file is a Python program setting a dictionary of the following format: ```python benchmarks = [ @@ -24,35 +24,72 @@ # List of measurements to make. 'measurements': [ - '<measurement type>/<event category>/<event name>', + { + 'name': <my_measurement>, + 'spec': <spec>, + }, + (...) ], }, ] ``` +The benchmark file may reference the `target_os` global that will be any of +('android', 'linux'), indicating the system on which the benchmarks are run. + +### Measurement specs + The following types of measurements are available: - - `time_until` - measures time until the first occurence of the specified event - - `avg_duration` - measures the average duration of all instances of the - specified event + - `time_until` + - `time_between` + - `avg_duration` + - `percentile_duration` + +`time_until` records the time until the first occurence of the targeted event. +The underlying benchmark runner records the time origin just before issuing the +connection call to the application being benchmarked. Results of `time_until` +measurements are relative to this time. Spec format: + +``` +'time_until/<category>/<event>' +``` + +`time_between` records the time between the first occurence of the first +targeted event and the first occurence of the second targeted event. Spec +format: + +``` +'time_between/<category1>/<event1>/<category2>/<event2>' +``` + +`avg_duration` records the average duration of all occurences of the targeted +event. Spec format: + +``` +'avg_duration/<category>/<event>' +``` + +`percentile_duration` records the value at the given percentile of durations of +all occurences of the targeted event. Spec format: + +``` +'percentile_duration/<category>/<event>/<percentile>' +``` + +where `<percentile>` is a number between 0.0 and 0.1. ## Caching The script runs each benchmark twice. The first run (**cold start**) clears caches of the following apps on startup: - - network_service.mojo - - url_response_disk_cache.mojo + - `network_service.mojo` + - `url_response_disk_cache.mojo` The second run (**warm start**) runs immediately afterwards, without clearing any caches. -## Time origin - -The underlying benchmark runner records the time origin just before issuing the -connection call to the application being benchmarked. Results of `time_until` -measurements are relative to this time. - ## Example For an app that records a trace event named "initialized" in category "my_app"
diff --git a/mojo_benchmark b/mojo_benchmark index 4a5c6fe..fc0258e 100755 --- a/mojo_benchmark +++ b/mojo_benchmark
@@ -20,47 +20,8 @@ _DESCRIPTION = """Runner for Mojo application benchmarks. |benchmark_list_file| has to be a valid Python program that sets a |benchmarks| -global variable, containing entries of the following form: - - { - 'name': '<name of the benchmark>', - 'app': '<url of the app to benchmark>', - 'shell-args': [], - 'duration': <duration in seconds>, - - # List of measurements to make. - 'measurements': [ - { - 'name': my_measurement, - 'spec': spec, - }, - (...) - ] - } - -For each measurement, 'name' is a label used for presentation purposes. 'spec' -defines the measurement. - -Available measurement types are: - -'time_until' - time until the first occurence of the targeted event. The spec -takes the following format: - - 'time_until/category/event' - -'avg_duration' - average duration of the targeted event. The spec takes the -following format: - - 'avg_duration/category/event' - -'percentile_duration' - value at the given percentile of the targeted event. The -spec takes the following format: - - 'percentile_duration/category/event/percentile' - -|benchmark_list_file| may reference the |target_os| global that will be any of -['android', 'linux'], indicating the system on which the benchmarks are to be -run. +dictionary. For description of the required format see +https://github.com/domokit/devtools/blob/master/docs/mojo_benchmark.md . """ _logger = logging.getLogger()