benchmark: add initial support for benchmark coverage - #54333
Conversation
Sorry, something went wrong.
|
Why don't we pass the function to benchmakr as a parameter to createBenchmark() like const bench = common.createBenchmark(main, {
n: [2e4],
benchmark_method: ['deepEqual', 'notDeepEqual', 'unequal_length'],
benchmark_module: 'assert',
}, {
combinationFilter: (p) => {
return p.strict === 1 || p.method === 'deepEqual';
},
});
and later access it as function run({ fn })
or something which resolves into node:assert 's function |
Sorry, something went wrong.
|
That requires changing all benchmarks and this is subject to developer errors. I tried to do it harmlessly (funny saying that while I'm monkey patching Module.require). But, possibly simpler than my approach. |
Sorry, something went wrong.
We can add an eslint rule (possibly @aduh95) to call that parameter inside bench.run() and bench.end() scope. |
Sorry, something went wrong.
This comment was marked as spam.
|
This pull request has been marked as stale due to 90 days of inactivity. |
Sorry, something went wrong.
Hey,
I'm opening it as a draft as there are a lot of loose ends to solve before considering such usage. This is an experiment related to research I'm conducting.
The idea is to generate a benchmark coverage for all of our exported modules. For instance, check if the functions exposed by require('node:fs') are covered in our benchmark suite (benchmark/fs/*.js). To achieve an intermediary goal I had to monkey-patch the Module.prototype.require before executing each benchmark (we run each one in a separate process) to return a singleton that basically manages the state of how many times that function was called.
I have tried to use our test_runner coverage for that, but it couldn't identify the location of built-in modules (require('node:*')) which is expected by the nature of a coverage tool. The other reason it didn't fit was the need to cover only the exported modules, not the lib/internal/* functions. The result I wanted to have was: "I need to know which functions/classes that are exposed to users do not contain a benchmark".
It's also worth it to mention that, a nested call is ignored in the benchmark report. Example:
node/lib/fs.js
Line 260 in b8a2550
Results
DetailsFull result: https://gist.github.com/RafaelGSS/66d33091560d61932b26d74af2fa8b82
Current limitations
This will result in:
This is unfortunate, as in reality only fs.existsSync is being measured. One theory that came to my mind to fix that is to intercept the bench.start calls and only then return the patched modules. I might try it later.
I'm certain there's a better way to approach this goal and fix the limitations. Hence, I'm opening it as a draft.
cc: @nodejs/benchmarking @nodejs/test_runner @lemire