test_runner: extend tag filter with boolean expression DSL by atlowChemi · Pull Request #63054 · nodejs/node · GitHub

Summary

Builds on #63221, which landed the tags option, inheritance, reporter
event payloads, and a literal-tag-name filter. This PR upgrades the
filter to a boolean expression DSL:

Grammar: and/&&, or/||, not/!, parentheses for grouping,
and * wildcards inside identifiers (bare * matches any tagged
test). Standard precedence (not > and > or); binary operators are
left-associative. Word forms (and/or/not) require whitespace
separation; punctuation forms do not.
--experimental-test-tag-filter and the testTagFilters run()
option now accept an expression rather than a literal tag name. The
flag is still repeatable; multiple expressions still AND together.
Untagged tests: any include expression evaluates false against an
empty tag set; not X evaluates true. So not flaky keeps every
untagged test, while db excludes them.
Tag value validation tightens: tag names may no longer contain
whitespace, operator characters (& | ! ( ) *), or the reserved
words and/or/not in any casing. This is a breaking change
relative to test_runner: add tags option and tag-name filter #63221 - acceptable since the feature is at Stability 1.0
(Early development).
Malformed expressions fail fast at parent-process startup before any
test file is spawned.

Prior art

Tag filtering with boolean composition is well-trodden in the JS
testing ecosystem. The expression syntax here is closest to Vitest's:

Vitest test tags - https://vitest.dev/guide/test-tags
mocha-tags - https://www.npmjs.com/package/mocha-tags
jest-runner-groups - https://www.npmjs.com/package/jest-runner-groups
jest-tags - https://github.com/gitbrik/jest-tags

Vitest additionally lets a tag carry per-test config overrides such as
timeout or retry (with priority resolution between overlapping
tags). That's intentionally out of scope for this PR - landing it
would commit to a separate tags: [{ name, timeout, retry, priority }]
config surface that's orthogonal to filtering. It can be layered on
later without breaking the union-inheritance semantics established in
#63221.

Review requested:

@nodejs/config
@nodejs/test_runner

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.34%. Comparing base (d996610) to head (5b911b0).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #63054      +/-   ##
==========================================
+ Coverage   90.30%   90.34%   +0.03%     
==========================================
  Files         751      751              
  Lines      249116   249525     +409     
  Branches    47043    47145     +102     
==========================================
+ Hits       224974   225429     +455     
+ Misses      15509    15502       -7     
+ Partials     8633     8594      -39

Files with missing lines	Coverage Δ
lib/internal/test_runner/runner.js	94.81% <100.00%> (+0.08%)	⬆️
lib/internal/test_runner/tag_filter.js	100.00% <100.00%> (ø)
lib/internal/test_runner/utils.js	66.17% <100.00%> (+0.18%)	⬆️

... and 34 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sweet! I'll try to carve out time on Saturday to reciew

Sorry for the delay. GitHub is having a bit of a stroke. Every time I try to add a comment to the review "Something went wrong" and I have to copy it, reload, paste, and try a couple more times.

I'll try to finish tomorrow (I got through 10 of 18 files, which doesn't include tag_filter.js).

General comment: this is getting to be a bit of a junk-drawer of arguments 😬

I agree, perhaps we could open a good first issue Issues that are suitable for first-time contributors. issue to handle this

GitHub is having a bit of a stroke.

@JakobJingleheimer github having issues? no way 😂

I'm not sure adding quotes helps here, to me at least that seems more confusing (e.g. git docs do not do that: https://git-scm.com/docs/git-log)

Suggested change

      
            ### `--experimental-test-tag-filter='<expr>'`
          
            ### `--experimental-test-tag-filter=<expr>`

I'm not too keen on creating a new language for that. Let's either use JS (e.g. --experimental-test-tag-filter='(tag) => tag.startsWith("foo-")'), RegEx (e.g. --experimental-test-tag-filter=/^foo-/), or let's not add it into core

Probably better if we have this conversation at: #63054 (comment)?

Having it in JS means we have a problem (can't serialize functions, so isolation: 'process' is not going to work straightforward). Using regex doesn't solve the problem of "is X or Y but not Z", which JS/DSL would..

After discussing this face to face with @atlowChemi and @benjamingr I see the value in this. Considered some alternatives for allowing usage of tags via CLI, without the need to use run directly. it seems this is the best option.
compared to passing a function or a regex, and due to the fact it exists in other langauages and other test runners in the echosystem

I was also convinced:

Every other framework (e.g. vitest) even in other languages (e.g. pytest) does this as well as userland libraries for test tags.
Regex looks at a single value, so it can't do "AND" like the array at the moment
If we allow "recursive" array + regex, we're effectively inventing a DSL just less readable than every other test runner
Function requires either a serializable function (which is a footgun, closures and stuff) or an expensive postMessage back and forth for evaluating it on the parent

So I'm +1 on this, it took me using the feature a bunch to understand this

or an expensive postMessage back and forth for evaluating it on the parent

This would also be expensive as it would mean the tag evaluation which currently happens synchronously in the Test constructor, would have to become async (as we would have to wait for the message event for the response) and would require some noisy changes to ensure the test can't start while the evaluation hasn't completed

@JakobJingleheimer @aduh95 are you OK with landing this? (see Moshe & Benjamin's recent comments)

CI: https://ci.nodejs.org/job/node-test-pull-request/75730/

CI: https://ci.nodejs.org/job/node-test-pull-request/75732/

Upgrades the experimental tag filter introduced by stage 1 to accept a boolean expression instead of a literal tag name. Grammar: `and`/`&&`, `or`/`||`, `not`/`!`, parentheses for grouping, and `*` wildcards inside identifiers. Standard precedence (`not > and > or`); binary operators are left-associative. Word forms require whitespace separation; punctuation forms do not. Untagged tests evaluate `false` for any include expression and `true` for `not X`, so excluding tags does not accidentally remove untagged tests. The flag and `testTagFilters` option are still repeatable; multiple expressions still AND together. Malformed expressions fail fast at the parent process at startup. Tag value validation tightens to reject whitespace, operator characters (`& | ! ( ) *`), and the reserved words `and`/`or`/`not` in any casing - a breaking change relative to the stage 1 ship, acceptable at Stability 1.0 (Early development). Signed-off-by: atlowChemi <chemi@atlow.co.il>

CI: https://ci.nodejs.org/job/node-test-pull-request/75809/

Landed in a674c5d

Upgrades the experimental tag filter introduced by stage 1 to accept a boolean expression instead of a literal tag name. Grammar: `and`/`&&`, `or`/`||`, `not`/`!`, parentheses for grouping, and `*` wildcards inside identifiers. Standard precedence (`not > and > or`); binary operators are left-associative. Word forms require whitespace separation; punctuation forms do not. Untagged tests evaluate `false` for any include expression and `true` for `not X`, so excluding tags does not accidentally remove untagged tests. The flag and `testTagFilters` option are still repeatable; multiple expressions still AND together. Malformed expressions fail fast at the parent process at startup. Tag value validation tightens to reject whitespace, operator characters (`& | ! ( ) *`), and the reserved words `and`/`or`/`not` in any casing - a breaking change relative to the stage 1 ship, acceptable at Stability 1.0 (Early development). Signed-off-by: atlowChemi <chemi@atlow.co.il> PR-URL: #63054 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Apr 30, 2026

atlowChemi added the test_runner Issues and PRs related to the test runner subsystem. label Apr 30, 2026

MoLow reviewed Apr 30, 2026

View reviewed changes

Comment thread doc/api/cli.md

JakobJingleheimer added this to Test Runner Board Apr 30, 2026

TheOneTheOnlyJJ reviewed May 1, 2026

View reviewed changes

Comment thread test/parallel/test-runner-tag-filter-cli.mjs Outdated

atlowChemi force-pushed the test_runner-tag-filter branch 3 times, most recently from 3591324 to 31a7737 Compare May 5, 2026 16:47

JakobJingleheimer reviewed May 6, 2026

View reviewed changes

atlowChemi force-pushed the test_runner-tag-filter branch from 31a7737 to 853f715 Compare May 7, 2026 19:13

atlowChemi mentioned this pull request May 10, 2026

test_runner: add tags option and tag-name filter #63221

Merged

atlowChemi force-pushed the test_runner-tag-filter branch 2 times, most recently from 131d524 to 3f583b5 Compare May 12, 2026 17:51

atlowChemi changed the title test_runner: add experimental tag-based test filtering test_runner: extend tag filter with boolean expression DSL May 12, 2026

aduh95 reviewed May 18, 2026

View reviewed changes

MoLow approved these changes Aug 9, 2026

View reviewed changes

benjamingr approved these changes Aug 9, 2026

View reviewed changes

atlowChemi force-pushed the test_runner-tag-filter branch from 3f583b5 to f66e962 Compare August 9, 2026 20:27

MoLow approved these changes Aug 9, 2026

View reviewed changes

atlowChemi force-pushed the test_runner-tag-filter branch from e4a46c1 to 7c3d927 Compare August 9, 2026 21:08

benjamingr approved these changes Aug 9, 2026

View reviewed changes

MoLow added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 10, 2026

github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 10, 2026

MoLow force-pushed the test_runner-tag-filter branch from 7c3d927 to 5b911b0 Compare August 12, 2026 20:18

MoLow approved these changes Aug 12, 2026

View reviewed changes

atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 13, 2026

MoLow approved these changes Aug 13, 2026

View reviewed changes

benjamingr approved these changes Aug 13, 2026

View reviewed changes

github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 13, 2026

MoLow added author ready PRs that have at least one approval, no outstanding review comments, and a CI started. commit-queue Add this label to land a pull request using GitHub Actions. labels Aug 13, 2026

nodejs-github-bot merged commit a674c5d into nodejs:main Aug 13, 2026
86 of 87 checks passed

github-project-automation Bot moved this to Done in Test Runner Board Aug 13, 2026

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Aug 13, 2026

atlowChemi deleted the test_runner-tag-filter branch August 13, 2026 19:41

github-actions Bot mentioned this pull request Aug 25, 2026

2026-08-26, Version 26.8.0 (Current) #65551

Merged

+              * Identifiers—any non-whitespace, non-operator characters. A literal
+                identifier matches a tag of the same value (case-insensitive).
+              * `*` wildcards inside an identifier match any sequence of characters.
+                A bare `*` matches any tagged test.
+              * Boolean operators with two equivalent forms:
+                * `and` / `&&`
+                * `or` / `||`
+                * `not` / `!`
+              * Parentheses for grouping.

		fail(nesting, loc, testNumber, name, details, directive, testId) {
		fail(nesting, loc, testNumber, name, details, directive, testId, tags) {

               This feature requires `--allow-worker` if used with the [Permission Model][].
-              ### `--experimental-test-tag-filter=<tag>`
+              ### `--experimental-test-tag-filter='<expr>'`

	### `--experimental-test-tag-filter='<expr>'`
	### `--experimental-test-tag-filter=<expr>`

test_runner: extend tag filter with boolean expression DSL by atlowChemi · Pull Request #63054 · nodejs/node · GitHub

Uh oh!

Uh oh!

test_runner: extend tag filter with boolean expression DSL - #63054

test_runner: extend tag filter with boolean expression DSL#63054nodejs-github-bot merged 1 commit intonodejs:mainnodejs/node:mainfrom atlowChemi:test_runner-tag-filteratlowChemi/node:test_runner-tag-filterCopy head branch name to clipboard

Conversation

atlowChemi commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Prior art

Uh oh!

nodejs-github-bot commented Apr 30, 2026

Uh oh!

Uh oh!

codecov Bot commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JakobJingleheimer commented Apr 30, 2026

Uh oh!

Uh oh!

JakobJingleheimer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JakobJingleheimer May 6, 2026

Choose a reason for hiding this comment

Uh oh!

atlowChemi May 6, 2026

Choose a reason for hiding this comment

Uh oh!

atlowChemi commented May 6, 2026

Uh oh!

aduh95 May 18, 2026

Choose a reason for hiding this comment

Uh oh!

aduh95 May 18, 2026

Choose a reason for hiding this comment

Uh oh!

atlowChemi May 19, 2026

Choose a reason for hiding this comment

Uh oh!

MoLow left a comment

Choose a reason for hiding this comment

Uh oh!

benjamingr commented Aug 9, 2026

Uh oh!

atlowChemi commented Aug 9, 2026

Uh oh!

atlowChemi commented Aug 9, 2026

Uh oh!

nodejs-github-bot commented Aug 10, 2026

Uh oh!

nodejs-github-bot commented Aug 10, 2026

Uh oh!

nodejs-github-bot commented Aug 13, 2026

Uh oh!

Uh oh!

nodejs-github-bot commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

test_runner: extend tag filter with boolean expression DSL#63054
nodejs-github-bot merged 1 commit into
nodejs:mainfrom
atlowChemi:test_runner-tag-filter

atlowChemi commented Apr 30, 2026 •
edited
Loading

codecov Bot commented Apr 30, 2026 •
edited
Loading