The following sections describe the agenda of the meeting and also serve as a place to put the corresponding meeting notes. After the meeting, these notes are moved to https://github.com/nixpkgs-architecture/meetings. This HedgeDoc link will be reused for all subsequent meetings.
- Matrix announcement: https://matrix.to/#/!djTaTBQyWEPRQxrPTb:nixos.org/$zUJuvmsVRAEiRyDXp183PJ_wABTUSHVx6Jr1qJp--Kc
- Recording link: https://www.youtube.com/watch?v=LHF077GXiz4
- Who records: @infinisil
- Who takes meeting notes: @infinisil
- Who leads the meeting: @infinisil
Member updates?
- @DavHau: Experimenting with module system based
stdenv.mkDerivationredesign: https://github.com/DavHau/build-modules- Only depends on nixpkgs lib
- Bootstraps by itself on some bootstrap binaries
- Build-time interpreter, replacable with something other than bash, tried micropython
- Should be easy to bootstrap (@Gytis: micropython does not seem to be easy to boostrap)
- Trying not to use env variables, only use json for communication between Nix and build and phases
- Uses module system, no more override functions, flexible
- Multiple build phases possible
- @infinisil: Dag? @DavHau: No still linear
- @DavHau: All phases can be rendered into individual derivations
- Output of one phase is the input of the next one
- @infinisil: Single derivation for lots of builds, multiple derivations for debugging
- @qyliss: Copying necessary?
- @DavHau: No, can write to a random path in the sandbox
- @qyliss: Most build phases involve modifying source root
- @John: Copy should be cheap too with a COW file system
- @infinisil: Generally files are just created, not modified, COW might work well
- @qyliss: COW often doesn't work because /nix/store is a different filesystem than build directory (usually /tmp). Could be fixed by changing Nix to have a build dir in /nix/store
- @John: We should make nixpkgs for the Nix we wish we had, since such problems are fixable
- @infinisil: There's already a lot of potential for nixpkgs improvements, let's not focus on changing Nix
- @John: Shouldn't get into the weeds of fixing Nix, but if we can improve the correctness of Nix for the benefit of nixpkgs let's go for it
- @ashkitten: Concerned if we ignore performance implications, Hydra as well, already very expensive. Performance regression tracking
- @qyliss: nixpkgs eval is already really slow, more so for cross. Not a lot of work for optimizing eval time. Only eval cache in Flakes, doesn't work for small differences. Is somebody concerned going forward?
- @Gytis: Splitting phases should be out of scope, adds a lot of complexity, there are better ways of dealing with this
- @John: @qyliss: Eval is cursed, store layer is more salvageable. Performance shouldn't get in the way of performance
- @qyliss: @Gytis: Splitting phases is very necessary, e.g. source only function, to get source with patches applied, useful for checking out source, AGPL compliance.
- @Gytis: Just out of scope of new builder infra, very complex, better ways to achieve it
- @qyliss: Second system without phases separation will leads us to never being able to do certain things
- @Dandellion: Use computed derivations? NixOS/rfcs#92
- @John: Can use those, but doesn't make it easier. More like a parallel, next version thing, very different spirit
- @DavHau: Phase splitting would allow better integration with tools like
nix-shell. Currently phases modified during build time. While learning Nix, even after understanding Nix code in nixpkgs, build time phases were still hard to understand. Reinventing this should be in scope - @Robert: Phases currently are inherently imperative. Should have something like phases. Tangent:
nix developarchitecturally sub-optimal, not working with sandbox, assumes specific implementation of nixpkgs - @tomberek: How far should the splitting go? Would be valuable, split up things too big. Improves derivation iteration. Could use that feature for production too. Could easily be abused
- @ashkitten: overlayfs instead of copying?
- @qyliss: Experimented with it, overlayfs works slightly differently from normal filesystems, breaks stuff in subtle ways, can't really fix ourselves, fundamental limitation
- @infinisil: Split derivation only at the root, create build code to simulate isolation between derivations within a single one
- @DavHau: Tried with dream2nix something like this, worked well
- @qyliss: Dynamic split/combined should also work, would be nice to do it without copying too much
- @DavHau: Hopes to be able to experiment with it
- @qyliss: Skeptical of worrying about performance later, especially for eval performance. Can't do much about it. Module system is especially bad for it. (@roberth, @Gytis: +1). Can't do anything requiring significantly more eval. Not sure about module system "spooky action at a distance"
- @David: Being able to access every option from every other
- @John: Ambient authority problem, hard to scale, against using the module system
- @Gytis: Module system per package
- @infinisil: Happens to some degree already with submodules
- @Gytis: Module system could be improved upon
- @infinisil: I don't think the module system can be sped up a lot, already fairly optimized
- @Gytis: this should improve modules systems performance NixOS/nix#5403
- @ashkitten: Regarding redesigning setup hooks. Relates to being able to use a single NixOS module multiple times, wanting to run a phase multiple times
- @John: At some point one should properly look into eval performance. As long as we end up with final in-store derivations results that look like we want. Less important what input looks like, output derivations more important, less rebuilds. Graph of derivations. Doing this makes a lot of things less daunting
- @roberth: Relates to https://github.com/nixpkgs-architecture/issues/issues/3
- @infinisil: @David used this as inspiration
- @Gytis: Implemented something just like that, https://github.com/gytis-ivaskevicius/nix-reinventing-the-wheel/ Can build something, no cross compilation or bootstrapping, but uses json, phases have indices, makes smaller phases easier and overrides easier
- @infinisil: Let's go over these redesigns next time and closely look at them