Add benchmark tracking workflow with GitHub Pages - #3028
Conversation
Sorry, something went wrong.
|
nice, this made me wonder if PHPBench could support JSON output by default (PR): ./bin/phpbench run examples/Benchmark/ --iterations=2 --report=github-action-benchmark --output=json --progress=none | jq
[
{
"name": "HashingBench#benchAlgos(0,0)",
"unit": "µs",
"value": "0.8",
"range": "+0.00%"
},
{
"name": "HashingBench#benchAlgos(1,0)",
"unit": "µs",
"value": "0.5",
"range": "+0.00%"
},
{
"name": "HashingBench#benchAlgos(2,0)",
"unit": "µs",
"value": "0.5",
"range": "+0.00%"
},
{
"name": "HashingBench#benchAlgos(3,0)",
"unit": "µs",
"value": "0.7",
"range": "+0.00%"
},
{
"name": "HashingBench#benchAlgos(4,0)",
"unit": "µs",
"value": "0.8",
"range": "+0.00%"
},
{
"name": "HashingBench#benchAlgos(5,0)",
"unit": "µs",
"value": "1.2",
"range": "-4.00%"
},
...
Using a currently custom report: 'github-action-benchmark' => [
'generator' => 'expression',
'cols' => [
'name' => 'format("%s#%s(%s)", first(benchmark_name), first(subject_name), first(variant_name))',
'unit' => 'coalesce(first(subject_time_unit), "microseconds")',
'value' => 'display_as_time(first(result_time_avg), first(subject_time_unit))',
'range' => 'format("%+.2f%%", first(result_comp_deviation))',
],
'aggregate' => ['benchmark_class', 'subject_name', 'variant_index'],
],
but maybe I could add it as a default 🤔 or maybe just a documented exeample. |
Sorry, something went wrong.
|
thanks, I'll try and replace the custom converson script with a PHPBench report 👍 |
Sorry, something went wrong.
|
@AJenbo I feel like I'm being stupd - but the action is failing for 3rd-party PRs: https://github.com/phpactor/phpactor/actions/runs/22880526045/job/66381997235?pr=3031 is that expected? nm, think we've figured it out: https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request_target |
Sorry, something went wrong.
|
@AJenbo I've now made the report generation possible natively in the newly tagged PHPBench 1.6.0: In phpbench.json add this custom report: {
// ...
"report.generators": {
"github-action-benchmark": {
"cols": [
"name",
"unit",
"value",
"range",
"extra"
],
"expressions": {
"name": "format('%s::%s%s', first(benchmark_name),first(subject_name),if(first(variant_name) != false, format(' (%s)', first(variant_name)), ''))",
"unit": "coalesce(time_unit(first(subject_time_unit), true),'µs')",
"value": "time_convert(mode(result_time_avg), 'us', time_unit(first(subject_time_unit)))",
"range": "format('± %.2f%%', rstdev(result_time_avg))",
"extra": "format('%d iterations, %d revs', first(variant_iterations), first(variant_revs))"
},
"aggregate": [
"suite_tag",
"benchmark_class",
"subject_name",
"variant_index"
],
"generator": "expression"
}
}
}
You can then generate the JSON report with: run: vendor/bin/phpbench run --progress=plain --report=github-action-benchmark --output=json > output.json |
Sorry, something went wrong.
|
Nice! |
Sorry, something went wrong.
This adds a GitHub Actions workflow that tracks PHPBench results over time
using benchmark-action/github-action-benchmark
and publishes them to GitHub Pages.
Here's what it looks like in practice on our project:
https://ajenbo.github.io/phpantom_lsp/dev/bench/
On push to master:
On pull requests:
Since PHPBench's XML output isn't directly compatible with the benchmark action,
a small PHP converter script (.github/phpbench_to_json.php) bridges the two formats.
The gh-pages branch is created automatically on the first run.
We use phpactor's benchmarks as the basis for performance testing in PHPantom and wanted to give something back. Hopefully this is useful for tracking regressions upstream too.