test_runner: support test plans - #52860
Conversation
Sorry, something went wrong.
|
Review requested:
|
Sorry, something went wrong.
|
IMO the t.plan() should also be settable via the test options |
Sorry, something went wrong.
Co-Authored-By: Marco Ippolito <marcoippolito54@gmail.com>
MoLow
left a comment
There was a problem hiding this comment.
Can you please elaborate on the use cases of this feature?
We already error if a test never ends and the event loop is drained - in what scenario should I worry not all assertions will be checked?
and what makes assertions different than any other piece of code inside the test?
Sorry, something went wrong.
It makes sure that an assertion is called inside a callback, when the callback might not be invoked.
test('make sure callback is invoked', (t) => {
t.plan(1)
server.listen(0, (r) => {
// assert something in here
t.ok(true)
});
});With plan you are sure than the callback was invoked and the assertion was performed, otherwise the test would pass anyways. |
Sorry, something went wrong.
|
I am -0.5 on adding this. |
Sorry, something went wrong.
|
Hey folks, Marco is working with me and I suggested that he works on this feature that is missing from the Node.js test runner and it's necessary in certain specific cases. The use case is fairly specific and it pertains callback-based APIs. When the callback is called and you need to make assertions there, there is no easy way to do that except telling the test that it should not end before X number of assertions have been made (bar a test timeout). Let's take this example from the Fastify docs: const tap = require('tap')
const buildFastify = require('./app')
tap.test('GET `/` route', t => {
t.plan(4)
const fastify = buildFastify()
// At the end of your tests it is highly recommended to call `.close()`
// to ensure that all connections to external services get closed.
t.teardown(() => fastify.close())
fastify.inject({
method: 'GET',
url: '/'
}, (err, response) => {
t.error(err)
t.equal(response.statusCode, 200)
t.equal(response.headers['content-type'], 'application/json; charset=utf-8')
t.same(response.json(), { hello: 'world' })
})
})Let's forget for a second that fastify.inject also has a promise API. Without t.plan(4) you have no easy way to make sure that the assertions in the callback have been invoked. E.g. if inject due to a bug didn't invoke the callback, this test would pass just fine if you didn't have a way to tell the test that you're expecting 4 assertions. I hope this clarifies why this feature is necessary. |
Sorry, something went wrong.
|
I think this feature is needed. @marco-ippolito can you add a test that verifies what happens if there were more assertion than the plan? |
Sorry, something went wrong.
|
This is missing a way for other assertion frameworks to tap into the test planning. If I recall correctly, the need for this was what stopped @cjihrig working on this feature. |
Sorry, something went wrong.
The node:assert module contains several top level APIs that do not make sense to expose as methods on t.assert. Examples include AssertionError and CallTracker. This commit removes such APIs from t.assert. Refs: #52860 PR-URL: #53049 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
The node:assert module contains several top level APIs that do not make sense to expose as methods on t.assert. Examples include AssertionError and CallTracker. This commit removes such APIs from t.assert. Refs: #52860 PR-URL: #53049 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
t.subtest -> t.test Refs: nodejs#52860
When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: nodejs#53169 Refs: nodejs#52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: #53169 Refs: #52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Co-Authored-By: Marco Ippolito <marcoippolito54@gmail.com> PR-URL: #52860 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) #52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) #52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) #52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: #53486
Co-Authored-By: Marco Ippolito <marcoippolito54@gmail.com> PR-URL: #52860 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) #52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) #52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) #52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: #53486
Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) #52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) #52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) #52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: #53486
Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) #52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) #52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) #52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: #53486
I picked up the work from this comment #44125 (comment) by @cjihrig to add support for test planning