Skip to content
This repository has been archived by the owner on Mar 11, 2024. It is now read-only.

Latest commit

 

History

History
95 lines (84 loc) · 7.21 KB

2022-07-21.md

File metadata and controls

95 lines (84 loc) · 7.21 KB

Discussing nixpkgs bootstrapping

Context: I asked @Ericson2314 whether he'd have time for a call to talk a bit about bootstrapping

  • pkgs/stdenv/generic: this is what actually makes the stdenv, including defines mkDerivation, setup.sh, it takes some inputs.

  • The other pkgs/stdenv/* defines the bootstrapping: many packages sets (aka stages), each of which has their own stdenv) that build upon each other, prior to the final package set (which is not defined here).

  • @infinisil: Which tools specifically are needed by stdenv.mkDerivation?

    • @john: Look at pkgs/stdenv/generic/default.nix
      • Contains cc and shell arguments
    • @john: There have to be more, sed, patchelf
    • pkgs/stdenv/common-path.nix
      • used as initialPath

  • @infinisil: stdenv isn't that important for redesign, let's look at bootstarpping

  • @john: Alright let's look at bootstrapping

    • default.nix: checks nix version

    • pkgs/top-level/impure.nix: fills in default configuration from impure sources (builtins.currenSystem, various environment).

    • pkgs/top-level/default.nix: starts doing work

      • Specifies local and optional cross platform, complete records not strings
      • Talks about stdenv, but we're not concerned about that
      • pkgs/stdenv/booter.nix isn't that important, used for cross and co. so we can delay tying the knot

    • pkgs/stdenv/default.nix: picks a chain of stages to bootstrap from

      • Everything except stagesLinux and stagesDarwin is pretty rotten

    • stagesLinux:

      • Starts with selecting the bootstrap tools
        • bootstrapp tools are all mixed together FHS style, not separate by purpose nix store style.
        • This is a headache, because it makes it easier later on to accidentally depend on more of the bootstrap tools than we mean to, once we start rebuilding things.
      • @john: There's bootstrap tools and there's a Nix-builtin sh (available via /bin/sh in the sandbox). We don't need the latter, only bootstrap tools would suffice as bootstrap tools already contains a shell.
      • @infinisil: But bootstrap tools relies on Hydra, how to do proper bootstrapping?
      • @john: Doesn't matter that it's built by Hydra, from Nix it's just a fetch
      • @infinisil: Why do we need this special build by Hydra?
      • @john: We want to control it because it's finnicky
        • Sucks, should have it more like guix.
        • Impure bootstrap sucks.
      • @infinisil: How would a pure bootstrap for nix look like?
      • @john: Same as guix, handrolled asm file, go from there
      • @alyssa: Reuse the one from guix, not sure about how integrated it is
      • @john: Decently integrated
      • @infinisil: If we wanna start from a good foundation we should look at adapting it to Nix
      • @qyliss: Guix is also supportive
      • Sponsors?
      • @infinisil: Guix only seems to support 32bit and 64bit linux for "Reduced Binary Seed Bootstrap" https://guix.gnu.org/manual/en/html_node/Reduced-Binary-Seed-Bootstrap.html
      • @qyliss: Would be the same as stdenv
      • @john: Darwin can continue working like it does now
      • @infinisil: Then let's make it possible to have reduced bootstrap if possible
      • @infinisil: Is it worth looking into reduced bootstrap when we can't get rid of bootstrap tools
      • @qyliss: Never possible to get rid of boostrap tools like this
      • @john: Might be possible. Linux could have open bootstrapping, Darwin might need closed source one, hidden implementation details etc.
      • @john: Could use this binary that runs on all the OSes
      • @john: Don't worry about Darwin
      • @qyliss: Darwin shouldn't hold innovations back
      • @infinisil: What is the shared interface between bootstraps for different platforms?
        • @john: There is no such thing, it depends on the platform, what tools you need on each
      • @infinisil: Are stages important?
        • @qyliss: Stages are a bit like overlays. When we start from nothing, at some point we have to build some packages from e.g. gcc$OLD_VERISON then gcc$NEW_VERSION
        • @infinisil: Could we use the nixpkgs fixed-point for this? E.g. pkgs.gcc6, then build pkgs.gcc7, etc. from that
        • @john: Can be possible. Cross compilation relies more on stages. Version solver
        • @infinisil: Goes into version cleanup in nixpkgs
        • @john: Debian has meta packages, "Need C compiler, don't care which one"
        • @infinisil: How about just doing this untanglinng from stages into nixpkgs fixed-point
        • @john: Makes new packages, leads to combinatory explosion of overrides
        • @qyliss: Less convenient, duplication, somebody needs to maintain the additional packages needed for bootstrapping
      • @infinisil: Let's look at an example, how to bootstrap pkgs.hello?
      • @john: Stages are more easy to understand from the top, from the bottom is hard to see
      • @infinisil: How does one write the stages? Maybe want to compile a C program -> look at latest GCC -> package all dependencies, go from there
      • @john: Could start clean, doing the guix thing, then splicing it into nixpkgs later
      • @john: Stages is useful for different ABI versions
      • @infinisil: Yeah nixpkgs is essentially instantiated multiple times with all the configurations needed, maybe parametrized by stdenv
      • @qyliss: No real consistency for what is stdenv what not. E.g. curses library could be per platform, but not in stdenv
      • @infinisil: What if we have something like pkgs.abiVersion and packages use that, then use extra overlays to change it. Maybe expensive?
      • @john: Probably works fine
      • @john: Logic programming language, specify version constraints
        • @infinisil: Separate issue
      • @john: Fixed point vs stages, don't care. Costly is mass rebuilds. Once it builds it's easy to refactor.
      • @john: Stdenv is fairly orthogonal to bootstrapping

    • @Gytis: targetPlatform? Can we get rid of it?

    • @qyliss: Needed for compilers and other things

    • @infinisil: buildPlatform applies to all packages, hostPlatform only applies to things you can run, and targetPlatform only applies to things that builds things that can run

    • @john: If we have to build cross compilers we need to somehow specify the target system

    • @john: We don't want to manually list all targetPlatforms

    • @qyliss: Any package can be target sensitive

    • @john: It's weird that you can buildDepends depend on a package that cares what the "next step" should be (is target sensative) when you don't. If you are not targe-sensative your deps should not be either.

    • @qyliss: Maybe it could make sense to have a restriction of a target-insensitive package not being able to depend on a target-sensitive package

    • @john: +1. Target sensativity "infected" downstream packages re rebuilds, and this should be tightly controlled.

    • @john: responding to others. Againm, just depend on what you need, no need for "stdenv" collection. So don't get confused/mislead by stdenv.*Platform, it's just the location where *Platform happens to reside today for arbitrary reasons.

@john: Good things come from following: https://en.wikipedia.org/wiki/Law_of_Demeter

BOTTOM