perf_hooks: add statistical hypothesis testing to histogram by jasnell · Pull Request #65416 · nodejs/node · GitHub

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

Review requested:

@nodejs/performance
@nodejs/tsc

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
src/histogram.cc	90.76%	4 Missing and 30 partials ⚠️
src/histogram-inl.h	86.36%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65416      +/-   ##
==========================================
+ Coverage   90.12%   90.14%   +0.02%     
==========================================
  Files         751      751              
  Lines      252536   253137     +601     
  Branches    47519    47631     +112     
==========================================
+ Hits       227598   228191     +593     
+ Misses      16250    16202      -48     
- Partials     8688     8744      +56

Files with missing lines	Coverage Δ
lib/internal/histogram.js	96.09% <100.00%> (+1.32%)	⬆️
src/histogram.h	68.42% <ø> (ø)
src/histogram-inl.h	84.45% <86.36%> (+2.21%)	⬆️
src/histogram.cc	84.62% <90.76%> (+2.83%)	⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

rslgtm

/cc @nodejs/performance

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus

Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus

CI: https://ci.nodejs.org/job/node-test-pull-request/76295/

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Landed in c28857e...bf67fdc

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Notable changes: benchmark: * (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416 crypto: * update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495 * (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411 diagnostics_channel: * mark TracingChannel as stable (Abdelrahman Awad) #64525 lib,src: * (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024 net: * (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974 perf_hooks: * (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416 repl: * add basic syntax highlighting (Aviv Keller) #64591 sqlite: * (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232 * (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232 util: * (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965 zlib: * (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339 PR-URL: #65551

jasnell requested review from Qard and mcollina August 20, 2026 06:25

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026

jasnell force-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71e Compare August 20, 2026 13:24

jasnell mentioned this pull request Aug 20, 2026

perf_hooks: add binary export/import to Histogram #65434

Open

mcollina approved these changes Aug 20, 2026

View reviewed changes