Commit 97fa075
test_runner: support running tests in process
This commit introduces a new --experimental-test-isolation flag
that, when set to 'none', causes the test runner to execute all
tests in the same process. By default, this is the main test
runner process, but if watch mode is enabled, it spawns a separate
process that runs all of the tests.
The default value of the new flag is 'process', which uses the
existing behavior of running each test file in its own child
process.
It is worth noting that when the isolation mode is 'none', globals
and all other top level logic (such as top level before() and after()
hooks) is shared among all files.
Co-authored-by: Moshe Atlow <moshe@atlow.co.il>
PR-URL: #53927
Fixes: #51548
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>1 parent 858b583 commit 97fa075
22 files changed
Lines changed: 746 additions & 206 deletions
File tree
- doc
- api
- lib/internal
- main
- test_runner
- src
- test
- fixtures/test-runner
- no-isolation
- snapshots
- parallel
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -1081,6 +1081,20 @@ generated as part of the test runner output. If no tests are run, a coverage | |||
| 1081 | 1081 | report is not generated. See the documentation on | |
| 1082 | 1082 | [collecting code coverage from tests][] for more details. | |
| 1083 | 1083 | ||
| 1084 | + ### `--experimental-test-isolation=mode` | ||
| 1085 | + | ||
| 1086 | + <!-- YAML | ||
| 1087 | + added: REPLACEME | ||
| 1088 | + --> | ||
| 1089 | + | ||
| 1090 | + > Stability: 1.0 - Early development | ||
| 1091 | + | ||
| 1092 | + Configures the type of test isolation used in the test runner. When `mode` is | ||
| 1093 | + `'process'`, each test file is run in a separate child process. When `mode` is | ||
| 1094 | + `'none'`, all test files run in the same process as the test runner. The default | ||
| 1095 | + isolation mode is `'process'`. This flag is ignored if the `--test` flag is not | ||
| 1096 | + present. See the [test runner execution model][] section for more information. | ||
| 1097 | + | ||
| 1084 | 1098 | ### `--experimental-test-module-mocks` | |
| 1085 | 1099 | ||
| 1086 | 1100 | <!-- YAML | |
@@ -2200,7 +2214,9 @@ added: | |||
| 2200 | 2214 | --> | |
| 2201 | 2215 | ||
| 2202 | 2216 | The maximum number of test files that the test runner CLI will execute | |
| 2203 | - concurrently. The default value is `os.availableParallelism() - 1`. | ||
| 2217 | + concurrently. If `--experimental-test-isolation` is set to `'none'`, this flag | ||
| 2218 | + is ignored and concurrency is one. Otherwise, concurrency defaults to | ||
| 2219 | + `os.availableParallelism() - 1`. | ||
| 2204 | 2220 | ||
| 2205 | 2221 | ### `--test-coverage-exclude` | |
| 2206 | 2222 | ||
@@ -2363,7 +2379,7 @@ added: v22.3.0 | |||
| 2363 | 2379 | ||
| 2364 | 2380 | > Stability: 1.0 - Early development | |
| 2365 | 2381 | ||
| 2366 | - Regenerates the snapshot file used by the test runner for [snapshot testing][]. | ||
| 2382 | + Regenerates the snapshot files used by the test runner for [snapshot testing][]. | ||
| 2367 | 2383 | Node.js must be started with the `--experimental-test-snapshots` flag in order | |
| 2368 | 2384 | to use this functionality. | |
| 2369 | 2385 | ||
@@ -3573,6 +3589,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12 | |||
| 3573 | 3589 | [snapshot testing]: test.md#snapshot-testing | |
| 3574 | 3590 | [syntax detection]: packages.md#syntax-detection | |
| 3575 | 3591 | [test reporters]: test.md#test-reporters | |
| 3592 | + [test runner execution model]: test.md#test-runner-execution-model | ||
| 3576 | 3593 | [timezone IDs]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones | |
| 3577 | 3594 | [tracking issue for user-land snapshots]: https://github.com/nodejs/node/issues/44014 | |
| 3578 | 3595 | [ways that `TZ` is handled in other environments]: https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -445,18 +445,26 @@ in the [test runner execution model][] section. | |||
| 445 | 445 | ||
| 446 | 446 | ### Test runner execution model | |
| 447 | 447 | ||
| 448 | - Each matching test file is executed in a separate child process. The maximum | ||
| 449 | - number of child processes running at any time is controlled by the | ||
| 450 | - [`--test-concurrency`][] flag. If the child process finishes with an exit code | ||
| 451 | - of 0, the test is considered passing. Otherwise, the test is considered to be a | ||
| 452 | - failure. Test files must be executable by Node.js, but are not required to use | ||
| 453 | - the `node:test` module internally. | ||
| 448 | + When process-level test isolation is enabled, each matching test file is | ||
| 449 | + executed in a separate child process. The maximum number of child processes | ||
| 450 | + running at any time is controlled by the [`--test-concurrency`][] flag. If the | ||
| 451 | + child process finishes with an exit code of 0, the test is considered passing. | ||
| 452 | + Otherwise, the test is considered to be a failure. Test files must be executable | ||
| 453 | + by Node.js, but are not required to use the `node:test` module internally. | ||
| 454 | 454 | ||
| 455 | 455 | Each test file is executed as if it was a regular script. That is, if the test | |
| 456 | 456 | file itself uses `node:test` to define tests, all of those tests will be | |
| 457 | 457 | executed within a single application thread, regardless of the value of the | |
| 458 | 458 | `concurrency` option of [`test()`][]. | |
| 459 | 459 | ||
| 460 | + When process-level test isolation is disabled, each matching test file is | ||
| 461 | + imported into the test runner process. Once all test files have been loaded, the | ||
| 462 | + top level tests are executed with a concurrency of one. Because the test files | ||
| 463 | + are all run within the same context, it is possible for tests to interact with | ||
| 464 | + each other in ways that are not possible when isolation is enabled. For example, | ||
| 465 | + if a test relies on global state, it is possible for that state to be modified | ||
| 466 | + by a test originating from another file. | ||
| 467 | + | ||
| 460 | 468 | ## Collecting code coverage | |
| 461 | 469 | ||
| 462 | 470 | > Stability: 1 - Experimental | |
@@ -933,7 +941,7 @@ the [`--experimental-test-snapshots`][] command-line flag. | |||
| 933 | 941 | Snapshot files are generated by starting Node.js with the | |
| 934 | 942 | [`--test-update-snapshots`][] command-line flag. A separate snapshot file is | |
| 935 | 943 | generated for each test file. By default, the snapshot file has the same name | |
| 936 | - as `process.argv[1]` with a `.snapshot` file extension. This behavior can be | ||
| 944 | + as the test file with a `.snapshot` file extension. This behavior can be | ||
| 937 | 945 | configured using the `snapshot.setResolveSnapshotPath()` function. Each | |
| 938 | 946 | snapshot assertion corresponds to an export in the snapshot file. | |
| 939 | 947 | ||
@@ -1239,6 +1247,9 @@ added: | |||
| 1239 | 1247 | - v18.9.0 | |
| 1240 | 1248 | - v16.19.0 | |
| 1241 | 1249 | changes: | |
| 1250 | + - version: REPLACEME | ||
| 1251 | + pr-url: https://github.com/nodejs/node/pull/53927 | ||
| 1252 | + description: Added the `isolation` option. | ||
| 1242 | 1253 | - version: v22.6.0 | |
| 1243 | 1254 | pr-url: https://github.com/nodejs/node/pull/53866 | |
| 1244 | 1255 | description: Added the `globPatterns` option. | |
@@ -1272,8 +1283,13 @@ changes: | |||
| 1272 | 1283 | * `inspectPort` {number|Function} Sets inspector port of test child process. | |
| 1273 | 1284 | This can be a number, or a function that takes no arguments and returns a | |
| 1274 | 1285 | number. If a nullish value is provided, each process gets its own port, | |
| 1275 | - incremented from the primary's `process.debugPort`. | ||
| 1276 | - **Default:** `undefined`. | ||
| 1286 | + incremented from the primary's `process.debugPort`. This option is ignored | ||
| 1287 | + if the `isolation` option is set to `'none'` as no child processes are | ||
| 1288 | + spawned. **Default:** `undefined`. | ||
| 1289 | + * `isolation` {string} Configures the type of test isolation. If set to | ||
| 1290 | + `'process'`, each test file is run in a separate child process. If set to | ||
| 1291 | + `'none'`, all test files run in the current process. **Default:** | ||
| 1292 | + `'process'`. | ||
| 1277 | 1293 | * `only`: {boolean} If truthy, the test context will only run tests that | |
| 1278 | 1294 | have the `only` option set | |
| 1279 | 1295 | * `setup` {Function} A function that accepts the `TestsStream` instance | |
@@ -1717,9 +1733,9 @@ added: v22.3.0 | |||
| 1717 | 1733 | ||
| 1718 | 1734 | * `fn` {Function} A function used to compute the location of the snapshot file. | |
| 1719 | 1735 | The function receives the path of the test file as its only argument. If the | |
| 1720 | - `process.argv[1]` is not associated with a file (for example in the REPL), | ||
| 1721 | - the input is undefined. `fn()` must return a string specifying the location of | ||
| 1722 | - the snapshot file. | ||
| 1736 | + test is not associated with a file (for example in the REPL), the input is | ||
| 1737 | + undefined. `fn()` must return a string specifying the location of the snapshot | ||
| 1738 | + snapshot file. | ||
| 1723 | 1739 | ||
| 1724 | 1740 | This function is used to customize the location of the snapshot file used for | |
| 1725 | 1741 | snapshot testing. By default, the snapshot filename is the same as the entry | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -185,6 +185,9 @@ Enable the experimental node:sqlite module. | |||
| 185 | 185 | .It Fl -experimental-test-coverage | |
| 186 | 186 | Enable code coverage in the test runner. | |
| 187 | 187 | . | |
| 188 | + .It Fl -experimental-test-isolation Ns = Ns Ar mode | ||
| 189 | + Configures the type of test isolation used in the test runner. | ||
| 190 | + . | ||
| 188 | 191 | .It Fl -experimental-test-module-mocks | |
| 189 | 192 | Enable module mocking in the test runner. | |
| 190 | 193 | . | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -21,7 +21,7 @@ markBootstrapComplete(); | |||
| 21 | 21 | ||
| 22 | 22 | const options = parseCommandLine(); | |
| 23 | 23 | ||
| 24 | - if (isUsingInspector()) { | ||
| 24 | + if (isUsingInspector() && options.isolation === 'process') { | ||
| 25 | 25 | process.emitWarning('Using the inspector with --test forces running at a concurrency of 1. ' + | |
| 26 | 26 | 'Use the inspectPort option to run with concurrency'); | |
| 27 | 27 | options.concurrency = 1; | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -334,4 +334,5 @@ module.exports = { | |||
| 334 | 334 | after: hook('after'), | |
| 335 | 335 | beforeEach: hook('beforeEach'), | |
| 336 | 336 | afterEach: hook('afterEach'), | |
| 337 | + startSubtestAfterBootstrap, | ||
| 337 | 338 | }; | |
0 commit comments