benchmark: add benchmarks for the test_runner by rluvaton · Pull Request #48931 · nodejs/node · GitHub

I'm trying to benchmark tests creation and execution

this is needed as we need to evaluate how a change in the test runner affects the performance:

test_runner: cleanup test timeout abort listener #48915 (comment)

Should we use finished on the reporter or this?

I mostly didn't understand what we are benchmarking. Can you leave some descriptive comments on what we are benchmarking? PS: The file names does not help either. (For example: plain-tests-as-fast-as-can)

cc @nodejs/performance @MoLow @cjihrig

I mostly didn't understand what we are benchmarking. Can you leave some descriptive comments on what we are benchmarking? PS: The file names does not help either. (For example: plain-tests-as-fast-as-can)

we want to compare how changes in test runner code affect its performance, but I agree naming can be improved

@rluvaton in regarding to #47945 (comment) - @cjihrig and myself were talking about what we want to benchmark - we probably also want to run tests with --test (probably one case should run a single file with --test and another should run multiple files (perhaps all of the test-runner fixture folder))
can you add those benchmarks to this pr? if not, Il do it in a follow-up PR

[...] and another should run multiple files (perhaps all of the test-runner fixture folder)

I don't think running the test-runner fixture folder is a good idea as it changes, while I think benchmarks are best when they are static

we probably also want to run tests with --test

I avoided that on purpose to reduce the scope of the benchmark, adding the --test as well will also benchmark the process and read files and more

[...] and another should run multiple files (perhaps all of the test-runner fixture folder)

I don't think running the test-runner fixture folder is a good idea as it changes, while I think benchmarks are best when they are static

it is ok since the benchmarks are compared before and after a change we want to understand the impact of. no baseline or benchmark is saved according to @cjihrig

we probably also want to run tests with --test

I avoided that on purpose to reduce the scope of the benchmark, adding the --test as well will also benchmark the process and read files and more

that is a good point, but I think that should only mean we should rely on this specific benchmark only if it is significantly different, not that we shouldn't test it at all

can you add those benchmarks to this pr? if not, Il do it in a follow-up PR

I prefer if you add those as you also need to add only and also I faced some problems with adding the --test flag, if it's really important to you I can put some time to fix those...

LGTM

Landed in f458e5b

PR-URL: nodejs#48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

PR-URL: nodejs#48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

PR-URL: nodejs/node#48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

Update the hooks benchmark so registered hooks perform the same anti-optimization assignment as test bodies instead of calling a noop. This keeps the measured hook path from using an empty callback. Links for more information on this actions: nodejs#48931 (comment) https://www.mail-archive.com/v8-users@googlegroups.com/msg05521.html Signed-off-by: Luan Muniz <luan@luanmuniz.com.br>

benchmark: add benchmarks for the test_runner

8700926

nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. test_runner Issues and PRs related to the test runner subsystem. labels Jul 26, 2023

remove todo

4781ca7

rluvaton commented Jul 26, 2023

View reviewed changes

remove 1K suites due to OOM in conjunction with 1K tests per suite

d9e9b1c

rluvaton mentioned this pull request Jul 26, 2023

test_runner: cleanup test timeout abort listener #48915

Merged

anonrig reviewed Jul 26, 2023

View reviewed changes

rename files

a67b01d

MoLow reviewed Jul 26, 2023

View reviewed changes

Comment thread benchmark/test_runner/global-concurrent-tests.js Outdated

MoLow reviewed Jul 26, 2023

View reviewed changes

Comment thread benchmark/test_runner/global-sequential-tests.js Outdated

MoLow reviewed Jul 26, 2023

View reviewed changes

Comment thread benchmark/test_runner/suite-tests.js Outdated

sync and async tests

82e20f7

rluvaton added 2 commits July 26, 2023 23:15

sync and async tests

1308c10

add concurrency options

929e0d2

MoLow mentioned this pull request Jul 26, 2023

test_runner: propagate only to test ancestors #48932

Closed

5 tasks

H4ad reviewed Jul 26, 2023

View reviewed changes

Comment thread benchmark/test_runner/global-concurrent-tests.js

rluvaton added 2 commits July 28, 2023 11:28

trigger GitHub Actions CI

dbb1380

avoid v8 optimization

994e575

MoLow approved these changes Jul 28, 2023

View reviewed changes

H4ad approved these changes Jul 28, 2023

View reviewed changes

atlowChemi approved these changes Jul 28, 2023

View reviewed changes

debadree25 approved these changes Jul 29, 2023

View reviewed changes

debadree25 added author ready PRs that have at least one approval, no outstanding review comments, and a CI started. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Jul 29, 2023

debadree25 added the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 29, 2023

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 29, 2023

nodejs-github-bot merged commit f458e5b into nodejs:main Jul 29, 2023

rluvaton deleted the add-test-runner-benchmarks branch July 29, 2023 18:09

RafaelGSS pushed a commit that referenced this pull request Aug 15, 2023

benchmark: add benchmarks for the test_runner

2608c9f

PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

UlisesGascon mentioned this pull request Aug 15, 2023

v20.6.0 proposal #49185

Merged

RafaelGSS pushed a commit that referenced this pull request Aug 17, 2023

benchmark: add benchmarks for the test_runner

771abcb

PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

targos pushed a commit that referenced this pull request Nov 27, 2023

benchmark: add benchmarks for the test_runner

0d0eb47

PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

targos mentioned this pull request Nov 28, 2023

v18.19.0 release proposal #50953

Merged

luanmuniz mentioned this pull request Jun 8, 2026

benchmark: add test runner hooks and options #63754

Merged

benchmark: add benchmarks for the test_runner by rluvaton · Pull Request #48931 · nodejs/node · GitHub

Uh oh!

Uh oh!

benchmark: add benchmarks for the test_runner - #48931

benchmark: add benchmarks for the test_runner#48931
nodejs-github-bot merged 9 commits into
nodejs:mainfrom
rluvaton:add-test-runner-benchmarks

rluvaton commented Jul 26, 2023 •
edited
Loading

Uh oh!

Uh oh!

rluvaton Jul 26, 2023

Uh oh!

anonrig left a comment

Uh oh!

anonrig commented Jul 26, 2023

Uh oh!

MoLow commented Jul 26, 2023 •
edited
Loading

Uh oh!

Uh oh!

Uh oh!

rluvaton commented Jul 26, 2023

Uh oh!

Uh oh!

Uh oh!

MoLow commented Jul 26, 2023

Uh oh!

Uh oh!

rluvaton commented Jul 28, 2023

Uh oh!

MoLow left a comment

Uh oh!

nodejs-github-bot commented Jul 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

+                  });
+                }
+                await Promise.all(promises);

benchmark: add benchmarks for the test_runner by rluvaton · Pull Request #48931 · nodejs/node · GitHub

Uh oh!

Uh oh!

benchmark: add benchmarks for the test_runner - #48931

benchmark: add benchmarks for the test_runner#48931nodejs-github-bot merged 9 commits intonodejs:mainnodejs/node:mainfrom rluvaton:add-test-runner-benchmarksrluvaton/node:add-test-runner-benchmarksCopy head branch name to clipboard

Conversation

rluvaton commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rluvaton Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

anonrig left a comment

Choose a reason for hiding this comment

Uh oh!

anonrig commented Jul 26, 2023

Uh oh!

MoLow commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rluvaton commented Jul 26, 2023

Uh oh!

Uh oh!

Uh oh!

MoLow commented Jul 26, 2023

Uh oh!

Uh oh!

rluvaton commented Jul 28, 2023

Uh oh!

MoLow left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

benchmark: add benchmarks for the test_runner#48931
nodejs-github-bot merged 9 commits into
nodejs:mainfrom
rluvaton:add-test-runner-benchmarks

rluvaton commented Jul 26, 2023 •
edited
Loading

MoLow commented Jul 26, 2023 •
edited
Loading