Continuous benchmarking with google benchmark and bencher.dev - #596
Open
biddisco wants to merge 9 commits into
Open
Continuous benchmarking with google benchmark and bencher.dev#596biddisco wants to merge 9 commits into
biddisco wants to merge 9 commits into
Conversation
Copy link
Copy Markdown
Collaborator
Sorry, something went wrong.
- Add CMake option IPPL_ENABLE_BENCHMARK (default OFF). - Fetch Google Benchmark v1.9.4 via FetchContent when enabled. - Add minimal BenchmarkSmoke executable to verify the integration. Part of continuous-benchmarking spec.
- Add cmake/AddIpplBenchmark.cmake defining add_ippl_benchmark(). - Support REPORT_TIMERS whitelist and IGNORE_TIMERS blacklist. - Generate <name>_timers.h with constexpr reportedTimers array. - Make REPORT_TIMERS and IGNORE_TIMERS mutually exclusive. - Register benchmark targets with ctest when BUILD_TESTING is ON. - Convert BenchmarkSmoke to use the helper.
- Add src/Utility/BenchmarkMetrics.h to bridge IpplTimings to bencher.dev BMF. - Compute latency (mean/min/max) and count per timer. - Filter emitted timers against the generated whitelist. - Update BenchmarkSmoke to use a Google Benchmark fixture, collect IpplTimings samples, and write BenchmarkSmoke.bmf.json. - Add ippl::initialize/finalize around the benchmark main. Part of continuous-benchmarking spec.
- Add demos/alpine/LandauDampingBench.cpp as the first real IPPL benchmark. - Use add_ippl_benchmark() with a REPORT_TIMERS whitelist (solve, pushVelocity, pushPosition, update, loadBalance). - Run pre_run() once as warmup, reset IpplTimings, then benchmark manager->advance() per Google Benchmark iteration. - Write BMF JSON via BenchmarkMetrics in TearDown. - Destroy the manager in TearDown before ippl::finalize() to avoid Kokkos deallocation-after-finalize warnings. - Consume demo-specific flags (--overallocate, --info) and pass the remaining args to Google Benchmark. Part of continuous-benchmarking spec.
- Add ci/scripts/install_bencher.sh to install the bencher.dev CLI. - Add ci/cscs/benchmark/benchmark.yml with per-platform benchmark jobs: - OpenMP on Eiger - CUDA on Daint GH200 - ROCm on Beverin MI300 - Each job builds with IPPL_ENABLE_BENCHMARK=ON, runs LandauDampingBench, saves JSON/BMF artifacts, and submits to bencher.dev only when BENCHER_API_TOKEN is set. - Jobs are non-blocking (allow_failure: true) and gated by IPPL_ENABLE_BENCHMARK variable. - Bencher project slug is set to ippl-opalx (TBD). Part of continuous-benchmarking spec.
Use IPPL_RUN_BENCHMARKS to control whether benchmark jobs run, keeping IPPL_ENABLE_BENCHMARK as the CMake build option only. This avoids confusion between the build-time and CI-time switches.
biddisco
force-pushed
the
continuous-benchmarking
branch
3 times, most recently
from
ca2a29b to
4cdffd8
Compare
August 28, 2026 15:33
- Add a 'benchmark' stage after 'ippl_test' in ci/cscs/common.yml.
- Add -DIPPL_ENABLE_BENCHMARK=ON to the release build jobs for OpenMP,
CUDA, and ROCm so LandauDampingBench is built with the normal build
artifact.
- Replace the standalone ci/cscs/benchmark/benchmark.yml with
platform-specific files:
- benchmark-common.yml: shared template and bencher submission logic
- benchmark-openmp.yml / benchmark-cuda.yml / benchmark-rocm.yml:
one job per platform that reuses the release build artifact
- Each platform top-level YAML (cscs-openmp.yml, cscs-gh200.yml,
cscs-mi300.yml) now includes its matching benchmark job file.
- Benchmark jobs depend on the release build job and the 4-rank release
test job via 'needs', and run with SLURM_NTASKS=4 so there is exactly
one benchmark data point per platform.
- Detect the LandauDampingBench binary at either bin/LandauDampingBench
(IPPL_USE_STANDARD_FOLDERS=ON) or demos/alpine/LandauDampingBench
(default layout) so both layouts work.
Copy link
Copy Markdown
Collaborator
Author
|
cscs-ci run cscs-ci-gh200 |
Sorry, something went wrong.
biddisco
force-pushed
the
continuous-benchmarking
branch
from
4cdffd8 to
33445be
Compare
August 28, 2026 15:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces continuous benchmarking for IPPL using Google Benchmark and bencher.dev.
What's added