test_runner: run single test file benchmark

[pmarchini]

With reference to #55723:

As I'm exploring the possibility of adding a new kind of isolation (worker thread), I think we should add at least one benchmark to measure the differences in overhead between various isolation methods.
For example, I would expect a worker thread to have less computational overhead than a process.
This benchmark is meant to trace and highlight these kinds of differences

[nodejs-github-bot]

Review requested:

• @nodejs/performance

[cjihrig]

As I'm exploring the possibility of adding a new kind of isolation (worker thread)

Is there any benefit to using worker threads in the test runner? My guess is that there isn't. Worker threads will be slower than no isolation, have less isolation than child processes, and a number of Node APIs either don't work, or work differently in worker threads.

[pmarchini]

As I'm exploring the possibility of adding a new kind of isolation (worker thread)

Is there any benefit to using worker threads in the test runner? My guess is that there isn't. Worker threads will be slower than no isolation, have less isolation than child processes, and a number of Node APIs either don't work, or work differently in worker threads.

I agree on the "somewhere in between offering limited benefits"

The only significant advantage I see is the overhead reduction.

I've experimented with a poc and the reduction seems to be decent but not incredible on a large test suite.

My idea would be to start with an isolation based on a "single worker" moving incrementally then to a "pool" of workers.
This could avoid even more overhead avoiding the creation of new workers.

[cjihrig]

I'm not opposed to supporting a pool, but I still think worker threads are not a great choice due to API differences.

FWIW, you might find these benchmarks interesting.

[pmarchini]

Thanks @cjihrig for the link! I've taken a quick look and I think I'll extend this bench 😁
Do you have any specific suggestions regarding this PR?

I'm not opposed to supporting a pool, but I still think worker threads are not a great choice due to API differences.

On this point, are you thinking about a pool of processes?

[cjihrig]

LGTM

On this point, are you thinking about a pool of processes?

I think that would be less surprising to users, albeit not as performant as using worker threads. No matter which approach you take, you'll also have to deal with certain edge cases. For example, only tests will require the --test-only flag. There might also be edge cases related to watch mode.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64400/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64404/