README.org

Package Sets

Example

A simple build pipeline to build a package-lock.json(v2/3) project. This is limited insofar as it doesn’t pass an nmDirCmd to non-root builders. Another snippet below shows the current way to deal with non-root packages that have install scripts ( honestly they’re rare ).

{ lib
, lockDir

, pkgsFor
, mkPkgEntSource
, mkNmDirPlockV3
, runCommandNoCC
, buildPkgEnt
, installPkgEnt

, nodejs  ? pkgsFor.nodejs-14_x
, flocoPackages ? {}
} @ prev: let

  # Leave these outside of the set to avoid clashing with Nixpkgs 
  callPackageWith  = autoArgs: pkgsFor.callPackageWith ( final // autoArgs );
  callPackagesWith = autoArgs: pkgsFor.callPackagesWith ( final // autoArgs );
  callPackage      = final.callPackageWith {};
  callPackages     = final.callPackagesWith {};

  final = prev // {
    # Override default of v16 used in Nixpkgs 
    nodejs = nodejs-14_x;

    metaSet = lib.metaSetFromPlockV3 { inherit lockDir; };
    mkNmDir = mkNmDirPlockV3 {
      # Packages will be pulled from here when their "key" ( "<IDENT>/<VERSION>" )
      # matches an attribute in the set.
      inherit (final) pkgSet;
      # Default settings. These are wiped out if you pass args again.
      copy = false;  # Symlink
      dev  = true;   # Include dev modules
    };

    # FIXME: handle subtrees
    doNmDir = { meta, ... } @ pkgEnt: let
      needsNm = meta.hasBuild || meta.hasInstallScript || meta.hasTest;
    in pkgEnt // ( lib.optionalAttrs needsNm {
      inherit (final) mkNmDir;
    } );

    doBuild = { meta, ... } @ pkgEnt:
      pkgEnt // ( lib.optionalAttrs meta.hasBuild {
        built   = final.buildPkgEnt pkgEnt;
        outPath = build'.built.outPath;
      } );

    doInstall = { meta, ... } @ pkgEnt:
      pkgEnt // ( lib.optionalAttrs meta.hasInstallScript {
        installed = final.installPkgEnt pkgEnt;
        outPath   = installed'.installed.outPath;
      } );

    doTest = { meta, ... } @ pkgEnt:
      pkgEnt // ( lib.optionalAttrs meta.hasTest {
        test = final.testPkgEnt pkgEnt;
      } );

    mkPkgEnt = path: {
      hasBuild
    , hasInstallScript
    , hasBin
    , hasTest ? false
    , ...
    } @ metaEnt: let
      simple = ! ( hasBuild || hasInstallScript || hasBin || hasTest );
      base   = ( mkPkgEntSource metaEnt ) // {
        # Add phony `mkNmDir' as a stub for non-root pkgs.
        #
        # FIXME: you can reuse the root's `node_modules/' dir until you have a
        # smarted solution here.
        # What you'll need to do is use `mkNmDir' to dump trees, and then `cd'
        # to the relevant project to run the build.
        # This is a PITA though because you have to manually arrange the build
        # order or toposort or something.
        #
        # Previously I was writing the ~7-10 packages that actually needed
        # installs to run by hand, honestly they run fine with just NaN in most
        # cases; but if people want "run in any project" then there's no way
        # around sitting down to write a routine to pull-down subtrees from
        # parent dirs to "refocus" a lock.
        mkNmDir = if path == "" then final.mkNmDir else ":";
      };
      done = ( doTest ( doInstall ( doBuild ( doNmDir base ) ) ) );
    in if simple then base // { prepared = base.source; outPath = base.source; }
                 else done;

    # Optionally you can merge these or cherry pick from external packages.
    pkgSet = builtins.mapAttrs final.mkPkgEnt metaSet.__entries;

    # Example of merging ( I recommend using `flocoOverlays' if this pkgSet wants to be used by other flakes )
    flocoPackages = flocoPackages // final.pkgSet;
  };

in final

Overall this pipeline is pretty short; the underlying builders are doing the heavy lifting, we’re just arranging them in our preferred order.

Nearly an Overlay

Since we’ve got this example here I want to point out something useful for “composing” a package sets together; in Nix we use a pattern called “overlays” ( see the ones defined in flake.nix for examples ).

If you made final an arg here at the top, and drop `let’ in the body, you have an overlay that you can compose with other package sets.

If you just want a recursive attrset then take this as it is.

If reading this just helped you understand what an overlay is: High Five!

Non-Root Install Scripts

I have dealt with this in most projects by manually writing a small snippet to drop in any directoryies I actually need. Honestly the vast majority of installScript run fine without modules as long as you provide node-gyp ( provided implicitly by genericInstall ) and ocassionally the “NaN” module ( you must provide this explicitly either with mkNmDir* or cp in a preConfigure hook does the trick ).

Those that don’t work with those on their own usually take only a few minutes to whip up a tree for. See the libtree documentation and mkNmDir documentation for more extensive examples, but theres a couple:

{ mkNmDir ? mkSourceTree, pkgSet, pkgsFor, flocoFetch, genericInstall }: let
  nmDirCmd = mkNmDir {
    # Use something from the package set.
    tree."node_modules/foo" = pkgSet."foo/1.0.0";
    # Use a local path
    tree."node_modules/bar" = flocoFetch { type = "path"; path = "./node_modules/bar"; };
    # Use a `flocoPackage' output from a flake. ( just an arbitrary field )
    tree."node_modules/baz" = ( builtins.getFlake "baz" ).flocoPackages.baz;
    ...
  };
in genericInstall {
  name = "junk-inst-1.0.0";
  src = flocoFetch { type = "path"; path = toString ./.; };
  inherit nmDirCmd;
  buildInputs = [pkgsFor.postgresql];
  ...
}

I am going to whip up some extensions to libtree soon to make this automatic but for now that’s the way it works.

If you have a package with a large number of deps, make a lock:

NPM_CONFIG_LOCKFILE_VERSION=3 npm i @foo/bar--package-lock-only --ignore-scripts;
jq '.packages[""]' > ./package.json;

This is nearly what genMeta does to generate metaSet files ( see section below ).

Generating metaSet using genMeta script

The top level flake exposes an “app” named genMeta which allows you to quickly generate and serialize a metaSet from a package descriptor ( “<NAME>@<VERSION>” string for example ). If saved to a file you can read it back into Nix using lib.libmeta.metaSetFromSerial to start using that package ( and all of its dependencies ) with pkg(Ent|Set) builders.

The serialized metaSet will be read back as if it had been created from a package-lock.json(V3) with some gap filling from the top level package’s source tree.

One important caveat here is that certain fields such at _meta.(plock|pjs|lockDir) will not be present ( because they pointed to a temporary directory ) and metaEnt records won’t have their entries.(pjs|plock) fields. This is fine because we’ve already scraped all the info we really care about from them.

Example using genMeta to produce a node-gyp build which depends on NaN

This is a real example pulled from a project I worked on which required @datadog/native-metrics@1.2.0 to be built with node-gyp; unlike most builds this one actually needed some members of its node_modules/ tree to compile successfully.

Rather than manually writing the ideal tree, and other metadata I just generated it and whipped up this minimal pkgEnt for my other projects to consume as a flake. For the purposes of this example I’ll use a plain default.nix build, but the pkgEnt defined here is “complete” insofar as it could be added to a flocoOverlays.pkgSet for consumption by other projects.

# Run:  nix run at-node-nix#genMeta -- @datadog/native-metrics@1.2.0 > meta.nix;
# Then define this `default.nix' and run `nix build --impure -f .;'
let
  at-node-nix = builtins.getFlake "at-node-nix";
  inherit (at-node-nix) lib;
  pkgsFor = at-node-nix.legacyPackages.${builtins.currentSystem};

  # Read our stashed metadata and deserialize it.
  metaSet = lib.libmeta.metaSetFromSerial ( import ./meta.nix );

  # Create a package entry with source/tarball members, and add the install to it.
  # This is a "full" `pkgEnt', which is superfulous for this example; but useful in
  # real projects so I went the extra mile.
  pkgEnt = let
    # The `meta.nix' file marked `_meta.rootKey = "@datadog/native-metrics/1.2.0"', but
    # I avoided hard coding that key here to make it easier to reuse this snippet.
    pkgEntSrc = pkgsFor.mkPkgEntSource metaSet.${metaSet._meta.rootKey};
    # The `installPkgEnt' builder is going to yank args from our `metaEnt' and `pkgEnt', and
    # then call `buildGyp' for us.
    installed = pkgsFor.installPkgEnt ( pkgEntSrc // {
      # Our generated metadata has stashed the ideal tree needed to create the `node_modules/' dir.
      nmDirCmd = pkgsFor.mkNmDirLinkCmd { tree = metaSet._meta.trees.prod; };
    } );
    # Manually extend the `pkgEnt' with our install, and since we're done with this package
    # we'll also set `prepared' and a top level `outPath' to follow good `pkgEnt' conventions.
    # Following these conventions is important to help other expressions consume this `pkgEnt'.
  in pkgEntSrc // { inherit installed; inherit (installed) outPath; prepared = installed; };

in pkgEnt.prepared  # Just an alias of the `installed' field which we can build with `nix build --impure -f .;'

In this case I know that none of the deps have builds or installs, and we don’t call the node-gyp-build binaries, so the unpacked tarballs are “good enough” as they are.

This may not be the case in your use case; but you can use this technique to chip away at packages that need special build recipes one at a time.

For clarity: you’ll use this exact pattern for builds that require non-Node.js buildInputs like libfoo.so and whatnot; you’ll just add them as args to installPkgEnt ( which is just a frontend over stdenv.mkDerivation ).

FIXME: _meta.tree has not been pruned for unsupported packages.

Either update this example or tweak the deserializer.