Parallel recipe-dir sub-builds

[ihnorton]

For large "forge"-style builds, it would be useful to allow recipe-dir sub-builds to run in parallel on a large build system. @-wolfv mentioned that "It would be relatively straightforward to do, the pieces are there" on Discord. Oversubscribing the build system is always a risk, but combining manual control over the number of concurrent sub-builds along with the ninja jobserver build would go a long way to preventing this.

[wolfv]

Does ninjas have a job server?
Our idea was to start all builds. Once we see one taking all the CPU we could send SIGHALT to all other processes and then pick them up again once the other build is done. Would need some experimentation though :)

[ihnorton]

Does ninjas have a job server?

There is a ninja soft fork from kitware with jobserver support (which is the source for the ninja package on pypi). There has been some recent activity on getting it upstreamed.

Once we see one taking all the CPU we could send SIGHALT to all other processes

😰

---

Another thought was to require all (cmake) build scripts to set the subninja prefix, and don't run the build command right away in the build script. Rattler could generate a top-level file to combine the subninja files at each level of the sorted build tree, and run one ninja -jN invocation against that. (but that starts to look like some of what bazel/buck2)

[wolfv]

Ah great, a job server was one of the pieces that would be great for something like this. But then you still end up, to some degree, with the problem of ninja, make, cargo, pip and others potentially running at the same time. But still .. could be great!

I would be very curious about trying things with Linux processes and stopping / resuming them. Not sure if it can work.

[traversaro]

I just launched a ros-humble rebuild on a 128-core server and it would be quite cool to have this at some point. : )