test: print failed JS/parallel tests by GeoffreyBooth · Pull Request #45960 · nodejs/node · GitHub

This is a small productivity improvement. When running make test, or specifically make test-parallel or make jstest, if there any failed JavaScript (or parallel-run) tests, the error output is printed but the only summary is a line like this:

[02:24|% 100|+ 3110|-   2]: Done

With this PR, in addition to the previous output the test runner will also print a list of the failed tests, like so:

[02:32|% 100|+ 3110|-   2]: Done
Failed tests:
out/Release/node --experimental-loader ./test/fixtures/es-module-loaders/loader-this-value-inside-hook-functions.mjs /node/test/parallel/test-loaders-this-value-inside-hook-functions.mjs
out/Release/node --experimental-loader ./test/fixtures/es-module-loaders/loader-unknown-builtin-module.mjs node/test/parallel/test-loaders-unknown-builtin-module.mjs

Again, this is in addition to the current output; the present full output for each test is preserved.

My reasons for adding this:

When there are many failing tests, it’s helpful to see a summary of all the failing tests to discern patterns of which groups of tests are failing. This cannot be done in the current output, where each failing test gets something like 10 to 100 lines of output.
The summary of commands makes it quick to re-run particular failed tests.

cc @nodejs/testing @nodejs/test_runner @nodejs/python

Huzzah!

/Cc @nodejs/testing

Can you add a test failure in this PR, and run a light CI to see it impacts it negatively?

Another tiny tweak that I think would make a huge improvement would be to add an empty line and/or horizontal line between test outputs (as it currently is, they are back to back, each block is nigh indistinguishable from the next).

CI: https://ci.nodejs.org/job/node-test-pull-request/48725/

Another tiny tweak that I think would make a huge improvement would be to add an empty line and/or horizontal line between test outputs (as it currently is, they are back to back, each block is nigh indistinguishable from the next).

I just tried it and I think two blank lines are ideal, because many of the tests have single blank lines within them (like after a stack trace, say) but rarely if ever two blank lines, so visually scanning down a lot of results the two blank lines help make the breaks stand out.

Can you add a test failure in this PR, and run a light CI to see it impacts it negatively?

This is a CI run with an intentionally failing test: https://ci.nodejs.org/job/node-test-pull-request/48725/

CI: https://ci.nodejs.org/job/node-test-pull-request/48728/

@cclauss I’ve added the type hint and responded to all of your suggestions. I believe this resolves your concerns; do you mind please removing your block?

CI: https://ci.nodejs.org/job/node-test-pull-request/48740/

Made requested changes

@cclauss Please let me know if you have any other notes.

Commit Queue failed

- Loading data for nodejs/node/pull/45960
✔  Done loading data for nodejs/node/pull/45960
----------------------------------- PR info ------------------------------------
Title      test: print failed JS/parallel tests (#45960)
   ⚠  Could not retrieve the email or name of the PR author's from user's GitHub profile!
Branch     GeoffreyBooth:print-failures -> nodejs:main
Labels     test, tools, python, author ready, commit-queue-squash
Commits    5
 - test: print failed JS/parallel tests
 - add type hint
 - Add newline between failed test outputs
 - Match style for variable names
 - Add import for backward compatibility
Committers 1
 - Geoffrey Booth 
PR-URL: https://github.com/nodejs/node/pull/45960
Reviewed-By: Jacob Smith 
Reviewed-By: Christian Clauss 
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/45960
Reviewed-By: Jacob Smith 
Reviewed-By: Christian Clauss 
--------------------------------------------------------------------------------
   ℹ  This PR was created on Sat, 24 Dec 2022 01:50:48 GMT
   ✔  Approvals: 2
   ✔  - Jacob Smith (@JakobJingleheimer): https://github.com/nodejs/node/pull/45960#pullrequestreview-1229549661
   ✔  - Christian Clauss (@cclauss): https://github.com/nodejs/node/pull/45960#pullrequestreview-1230212097
   ✖  Last GitHub CI failed
   ℹ  Green GitHub CI is sufficient
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu

https://github.com/nodejs/node/actions/runs/3783385909

Landed in eb4c83f

PR-URL: #45960 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Christian Clauss <cclauss@me.com>

GeoffreyBooth added test Issues and PRs related to the tests. python PRs and issues that require attention from people who are familiar with Python. labels Dec 24, 2022

GeoffreyBooth requested review from JakobJingleheimer and aduh95 December 24, 2022 01:50

nodejs-github-bot added the tools Issues and PRs related to the tools directory. label Dec 24, 2022

JakobJingleheimer approved these changes Dec 24, 2022

View reviewed changes

cclauss previously requested changes Dec 24, 2022

View reviewed changes

Comment thread tools/test.py Outdated

Comment thread tools/test.py Outdated

cclauss reviewed Dec 24, 2022

View reviewed changes

Comment thread tools/test.py Outdated

cclauss reviewed Dec 24, 2022

View reviewed changes

Comment thread tools/test.py Outdated

GeoffreyBooth added 2 commits December 24, 2022 15:36

test: print failed JS/parallel tests

8c6c1fb

add type hint

639e1a8

GeoffreyBooth force-pushed the print-failures branch from fa7ce89 to 5254d06 Compare December 24, 2022 23:42

GeoffreyBooth requested a review from cclauss December 24, 2022 23:42

Add newline between failed test outputs

a645ef6

GeoffreyBooth force-pushed the print-failures branch from 5254d06 to a645ef6 Compare December 25, 2022 00:11

Match style for variable names

fa70072

nodejs deleted a comment from nodejs-github-bot Dec 25, 2022

github-actions Bot mentioned this pull request Dec 26, 2022

CI Reliability 2022-12-26 nodejs/reliability#469

Open

8 tasks

cclauss reviewed Dec 26, 2022

View reviewed changes

Comment thread tools/test.py Outdated

Add import for backward compatibility

193fdc0

cclauss approved these changes Dec 26, 2022

View reviewed changes

nodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels Dec 26, 2022

GeoffreyBooth added commit-queue Add this label to land a pull request using GitHub Actions. and removed commit-queue-failed An error occurred while landing this pull request using GitHub Actions. labels Dec 27, 2022

github-actions Bot mentioned this pull request Dec 27, 2022

CI Reliability 2022-12-27 nodejs/reliability#470

Open

8 tasks

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Dec 27, 2022

nodejs-github-bot merged commit eb4c83f into nodejs:main Dec 27, 2022

GeoffreyBooth deleted the print-failures branch December 27, 2022 04:03

targos pushed a commit that referenced this pull request Jan 1, 2023

test: print failed JS/parallel tests

50ddc7c

PR-URL: #45960 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Christian Clauss <cclauss@me.com>

github-actions Bot mentioned this pull request Jan 2, 2023

CI Reliability 2023-01-02 nodejs/reliability#476

Open

22 tasks

RafaelGSS mentioned this pull request Jan 2, 2023

v19.4.0 proposal #46061

Merged

github-actions Bot mentioned this pull request Jan 3, 2023

CI Reliability 2023-01-03 nodejs/reliability#477

Open

23 tasks

RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023

test: print failed JS/parallel tests

bee60bc

PR-URL: #45960 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Christian Clauss <cclauss@me.com>

RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023

test: print failed JS/parallel tests

0f5a145

PR-URL: #45960 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Christian Clauss <cclauss@me.com>

juanarbol pushed a commit that referenced this pull request Jan 26, 2023

test: print failed JS/parallel tests

e9b1879

PR-URL: #45960 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Christian Clauss <cclauss@me.com>

juanarbol mentioned this pull request Jan 28, 2023

V18.14.0 proposal #46396

Merged

juanarbol pushed a commit that referenced this pull request Jan 31, 2023

test: print failed JS/parallel tests

4d6dd10

PR-URL: #45960 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Christian Clauss <cclauss@me.com>

test: print failed JS/parallel tests by GeoffreyBooth · Pull Request #45960 · nodejs/node · GitHub

Uh oh!

Uh oh!

test: print failed JS/parallel tests - #45960

test: print failed JS/parallel tests#45960nodejs-github-bot merged 5 commits intonodejs:mainnodejs/node:mainfrom GeoffreyBooth:print-failuresGeoffreyBooth/node:print-failuresCopy head branch name to clipboard

Conversation

GeoffreyBooth commented Dec 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JakobJingleheimer left a comment

Choose a reason for hiding this comment

Uh oh!

aduh95 commented Dec 24, 2022

Uh oh!

aduh95 commented Dec 24, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JakobJingleheimer commented Dec 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented Dec 24, 2022

Uh oh!

GeoffreyBooth commented Dec 25, 2022

Uh oh!

GeoffreyBooth commented Dec 25, 2022

Uh oh!

nodejs-github-bot commented Dec 25, 2022

Uh oh!

GeoffreyBooth commented Dec 26, 2022

Uh oh!

Uh oh!

nodejs-github-bot commented Dec 26, 2022

Uh oh!

GeoffreyBooth commented Dec 26, 2022

Uh oh!

nodejs-github-bot commented Dec 26, 2022

Uh oh!

nodejs-github-bot commented Dec 27, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

test: print failed JS/parallel tests#45960
nodejs-github-bot merged 5 commits into
nodejs:mainfrom
GeoffreyBooth:print-failures

GeoffreyBooth commented Dec 24, 2022 •
edited
Loading

JakobJingleheimer commented Dec 24, 2022 •
edited
Loading