FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

doc: update WPT test runner README.md · nodejs/node@8126d1c · GitHub

/ node Public

Commit 8126d1c

Browse files
panva
authored and
aduh95
committed
doc: update WPT test runner README.md
Signed-off-by: Filip Skokan <panva.ip@gmail.com> PR-URL: #62680 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
1 parent f00cdab commit 8126d1c

1 file changed

Lines changed: 145 additions & 116 deletions

File tree

‎test/wpt/README.md‎

Lines changed: 145 additions & 116 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,27 @@
11
# Web Platform Tests
22

3-
The tests here are drivers for running the [Web Platform Tests][].
4-
5-
See [`test/fixtures/wpt/README.md`][] for a hash of the last
6-
updated WPT commit for each module being covered here.
7-
8-
See the json files in [the `status` folder](./status) for prerequisites,
9-
expected failures, and support status for specific tests in each module.
10-
11-
Currently there are still some Web Platform Tests titled `test-whatwg-*`
12-
under `test/parallel` that have not been migrated to be run with the
13-
WPT harness and have automatic updates. There are also a few
14-
`test-whatwg-*-custom-*` tests that may need to be upstreamed.
15-
This folder covers the tests that have been migrated.
3+
This directory contains test runners that execute upstream
4+
[Web Platform Tests][] against Node.js using the WPT harness.
5+
The actual test files live in `test/fixtures/wpt`, a subset of the
6+
upstream WPT repository containing only the modules relevant to Node.js.
7+
Each module is updated independently using [git node wpt][], so
8+
different modules may be pinned to different upstream commits.
9+
10+
Each module has a status file in the [`status` folder](./status) that
11+
declares build requirements, expected failures, and tests to skip.
12+
See [`test/fixtures/wpt/README.md`][] for the pinned WPT commit
13+
hashes for each module.
1614

1715
<a id="add-tests"></a>
1816

1917
## How to add tests for a new module
2018

2119
### 1. Create a status file
2220

23-
For example, to add the URL tests, add a `test/wpt/status/url.json` file.
21+
For example, to add the URL tests, add a `test/wpt/status/url.cjs` file.
2422

25-
In the beginning, it's fine to leave an empty object `{}` in the file if
26-
it's not yet clear how compliant the implementation is,
23+
In the beginning, it's fine to leave an empty object `module.exports = {}`
24+
in the file if it's not yet clear how compliant the implementation is,
2725
the requirements and expected failures can be figured out in a later step
2826
when the tests are run for the first time.
2927

@@ -39,7 +37,7 @@ cd /path/to/node/project
3937
git node wpt url
4038
```
4139

42-
### 3. Create the test driver
40+
### 3. Create the test runner
4341

4442
For example, for the URL tests, add a file `test/wpt/test-url.js`:
4543

@@ -50,21 +48,49 @@ const { WPTRunner } = require('../common/wpt');
5048

5149
const runner = new WPTRunner('url');
5250

53-
// Set Node.js flags required for the tests.
54-
runner.setFlags(['--expose-internals']);
55-
56-
// Set a script that will be executed in the worker before running the tests.
57-
runner.setInitScript(`
58-
const { internalBinding } = require('internal/test/binding');
59-
const { DOMException } = internalBinding('messaging');
60-
global.DOMException = DOMException;
61-
`);
62-
51+
runner.pretendGlobalThisAs('Window');
6352
runner.runJsTests();
6453
```
6554

66-
This driver is capable of running the tests located in `test/fixtures/wpt/url`
67-
with the WPT harness while taking the status file into account.
55+
The runner loads the tests from `test/fixtures/wpt/url`, applies the
56+
status rules from `test/wpt/status/url.cjs`, and runs them using
57+
worker threads.
58+
59+
#### `new WPTRunner(path[, options])`
60+
61+
* `path` {string} Relative path of the WPT module
62+
(e.g. `'url'`, `'html/webappapis/timers'`).
63+
* `options` {Object}
64+
* `concurrency` {number} Number of tests to run in parallel.
65+
Defaults to `os.availableParallelism() - 1`. Set to `1` for tests
66+
that require sequential execution (e.g. web-locks, webstorage).
67+
68+
#### `runner.setFlags(flags)`
69+
70+
* `flags` {string\[]} Node.js CLI flags passed to each worker thread
71+
(e.g. `['--expose-internals']`).
72+
73+
#### `runner.setInitScript(script)`
74+
75+
* `script` {string} JavaScript code executed in the worker before
76+
the tests run. Useful for setting up globals needed by the tests.
77+
78+
#### `runner.setScriptModifier(modifier)`
79+
80+
* `modifier` {Function} A callback `(meta) => void` invoked for each
81+
script before it is run in the worker. `meta` is an object with
82+
`code` {string} and `filename` {string} properties that can be
83+
mutated.
84+
85+
#### `runner.pretendGlobalThisAs(name)`
86+
87+
* `name` {string} Currently only `'Window'` is supported. Sets up
88+
`globalThis.Window` so that WPT tests checking the global scope
89+
type work correctly.
90+
91+
#### `runner.runJsTests()`
92+
93+
Starts running the tests. Must be called last, after all configuration.
6894

6995
### 4. Run the tests
7096

@@ -76,26 +102,28 @@ tools/test.py wpt/test-url
76102
```
77103

78104
To run a specific test in WPT, for example, `url/url-searchparams.any.js`,
79-
pass the file name as argument to the corresponding test driver:
105+
pass the file name as argument to the corresponding test runner:
80106

81107
```bash
82108
node test/wpt/test-url.js url-searchparams.any.js
83109
```
84110

85111
If there are any failures, update the corresponding status file
86-
(in this case, `test/wpt/status/url.json`) to make the test pass.
112+
(in this case, `test/wpt/status/url.cjs`) to make the test pass.
87113

88114
For example, to mark `url/url-searchparams.any.js` as expected to fail,
89-
add this to `test/wpt/status/url.json`:
90-
91-
```json
92-
"url-searchparams.any.js": {
93-
"fail": {
94-
"expected": [
95-
"test name in the WPT test case, e.g. second argument passed to test()"
96-
]
97-
}
98-
}
115+
add this to `test/wpt/status/url.cjs`:
116+
117+
```js
118+
module.exports = {
119+
'url-searchparams.any.js': {
120+
fail: {
121+
expected: [
122+
'test name in the WPT test case, e.g. second argument passed to test()',
123+
],
124+
},
125+
},
126+
};
99127
```
100128

101129
See [Format of a status file](#status-format) for details.
@@ -110,90 +138,78 @@ The tests can be updated in a way similar to how they are added.
110138
Run Step 2 and Step 4 of [adding tests for a new module](#add-tests).
111139

112140
The [git node wpt][] command maintains the status of the local
113-
WPT subset, if no files are updated after running it for a module,
114-
the local subset is up to date and there is no need to update them
115-
until they are changed in the upstream.
141+
WPT subset. If no files are updated after running it for a module,
142+
the local subset is up to date and there is no need to create a PR.
143+
When files are updated, run the tests and update the status file to
144+
account for any new failures or passes before submitting.
116145

117-
## How it works
146+
## Daily WPT report
118147

119-
Note: currently this test suite only supports `.js` tests. There is
120-
ongoing work in the upstream to properly split out the tests into files
121-
that can be run in a shell environment like Node.js.
148+
A [GitHub Actions workflow][] runs every night and uploads results to
149+
[wpt.fyi][]. It tests all active Node.js release lines and the latest
150+
nightly build against the WPT `epochs/daily` branch, which is a daily
151+
snapshot of the upstream WPT repository.
122152

123-
### Getting the original test files and harness from WPT
124-
125-
The original files and harness from WPT are downloaded and stored in
126-
`test/fixtures/wpt`.
127-
128-
The [git node wpt][] command automate this process while maintaining a map
129-
containing the hash of the last updated commit for each module in
130-
`test/fixtures/wpt/versions.json` and [`test/fixtures/wpt/README.md`][].
131-
It also maintains the LICENSE file in `test/fixtures/wpt`.
132-
133-
### Loading and running the tests
134-
135-
Given a module, the `WPTRunner` class in [`test/common/wpt`](../common/wpt.js)
136-
loads:
137-
138-
* `.js` test files (for example, `test/common/wpt/url/*.js` for `url`)
139-
* Status file (for example, `test/wpt/status/url.json` for `url`)
140-
* The WPT harness
141-
142-
Then, for each test, it creates a worker thread with the globals and mocks,
143-
sets up the harness result hooks, loads the metadata in the test (including
144-
loading extra resources), and runs all the tests in that worker thread,
145-
skipping tests that cannot be run because of lack of dependency or
146-
expected failures.
153+
Unlike the pinned fixtures used in CI, this workflow replaces
154+
`test/fixtures/wpt` with the full `epochs/daily` checkout so that
155+
results reflect the latest upstream tests. Results can be viewed on
156+
the [wpt.fyi dashboard][].
147157

148158
<a id="status-format"></a>
149159

150160
## Format of a status file
151161

152-
```json
153-
{
154-
"something.scope.js": { // the file name
155-
// Optional: If the requirement is not met, this test will be skipped
156-
"requires": ["small-icu"], // supports: "small-icu", "full-icu", "crypto"
157-
158-
// Optional: the entire file will be skipped with the reason printed
159-
"skip": "explain why we cannot run a test that's supposed to pass",
160-
161-
// Optional: failing tests
162-
"fail": {
163-
"note": "You may leave an optional arbitrary note e.g. with TODOs",
164-
"expected": [
165-
"test name in the WPT test case, e.g. second argument passed to test()",
166-
"another test name"
162+
The status file can be either a `.json` file or a `.cjs` module that exports
163+
the same object. Using CJS allows for conditional logic and regular
164+
expressions, which JSON does not support.
165+
166+
```js
167+
module.exports = {
168+
'something.scope.js': { // the file name
169+
// Optional: If the requirement is not met, this test will be skipped.
170+
// Supported values:
171+
// 'small-icu' - requires at least small-icu intl support
172+
// 'full-icu' - requires full-icu intl support
173+
// 'crypto' - requires crypto (OpenSSL) support
174+
// 'inspector' - requires the inspector to be available
175+
requires: ['small-icu'],
176+
177+
// Optional: the entire file will be skipped with the reason printed.
178+
skip: 'explain why we cannot run a test that is supposed to pass',
179+
180+
// Optional: failing tests.
181+
fail: {
182+
// Tests that are expected to fail consistently.
183+
expected: [
184+
'test name in the WPT test case, e.g. second argument passed to test()',
185+
'another test name',
167186
],
168-
"flaky": [
169-
"flaky test name"
170-
]
171-
}
172-
}
173-
}
187+
// Tests that fail intermittently. These are treated as expected
188+
// failures but are not flagged as unexpected passes when they
189+
// succeed.
190+
flaky: [
191+
'flaky test name',
192+
],
193+
},
194+
},
195+
};
174196
```
175197

198+
A test should be marked with `skip` when it cannot be run at all, for
199+
example, because it depends on a browser-only Web API or a harness feature
200+
that has not been ported to the Node.js runner. Use `fail` instead when
201+
the test can run but produces incorrect results due to an implementation
202+
bug or missing feature.
203+
176204
### Skipping individual subtests
177205

178206
To skip specific subtests within a file (rather than skipping the entire file),
179-
use `skipTests` with an array of exact test names:
180-
181-
```json
182-
{
183-
"something.scope.js": {
184-
"skipTests": [
185-
"exact test name to skip"
186-
]
187-
}
188-
}
189-
```
190-
191-
When the status file is a CJS module, regular expressions can also be used:
207+
use `skipTests` with an array of exact test names or regular expressions:
192208

193209
```js
194210
module.exports = {
195211
'something.scope.js': {
196-
'skipTests': [
212+
skipTests: [
197213
'exact test name to skip',
198214
/regexp pattern to match/,
199215
],
@@ -205,18 +221,31 @@ Skipped subtests are reported as `[SKIP]` in the output, recorded as `NOTRUN`
205221
in the WPT report, and counted separately in the summary line.
206222

207223
This is useful for skipping a particular subtest that crashes the runner,
208-
which would otherwise prevent the rest of the file from being run. When using
209-
CJS status files, this also enables conditionally skipping slow or
210-
resource-heavy subtests in CI on specific architectures.
224+
which would otherwise prevent the rest of the file from being run. Using CJS
225+
status files also enables conditionally skipping slow or resource-heavy
226+
subtests in CI on specific architectures.
227+
228+
### Wildcard patterns in file names
229+
230+
File name keys can include a `*` character to match multiple test files
231+
with a single entry. For example, to skip all `.window.js` tests:
211232

212-
A test may have to be skipped because it depends on another irrelevant
213-
Web API, or certain harness has not been ported in our test runner yet.
214-
In that case it needs to be marked with `skip` instead of `fail`.
233+
```js
234+
module.exports = {
235+
'*.window.js': {
236+
skip: 'window tests are not relevant for Node.js',
237+
},
238+
};
239+
```
215240

216-
The status file may optionally also be a CJS module that exports the object.
217-
This allows for more complex logic to be used to determine the expected status
218-
of a test.
241+
The `*` is converted to a `.*` regular expression, so `"subdir/*.any.js"`
242+
would match all `.any.js` files under the `subdir` directory. A test file
243+
can match multiple rules (both an exact match and one or more wildcard
244+
patterns); all matched rules are merged.
219245

246+
[GitHub Actions workflow]: ../../.github/workflows/daily-wpt-fyi.yml
220247
[Web Platform Tests]: https://github.com/web-platform-tests/wpt
221248
[`test/fixtures/wpt/README.md`]: ../fixtures/wpt/README.md
222249
[git node wpt]: https://github.com/nodejs/node-core-utils/blob/HEAD/docs/git-node.md#git-node-wpt
250+
[wpt.fyi]: https://wpt.fyi
251+
[wpt.fyi dashboard]: https://wpt.fyi/results/?label=master&label=experimental&product=node.js&product=chrome&product=firefox&product=safari&product=ladybird&product=servo&q=node.js%3A%21missing

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL