README.org

node_modules directory builders

FIXME: This file is out of date

mkNmDirCmdWith has changed the prototype for add[Bin]Cmd, and mkPkgEntSource has also changed. This file needs to be updated.

Builders

mkNmDirCmdWith

Base definition from which all others are built.

Copies or links all modules from Nix store to path indicated by $node_modules_path.

Overview

Given an NM tree represented as an attrset of { <NM-PATH> = PKG; } entries, produce a shell script which builds a node_modules/ directory.

The PKG value can be a pkgEnt attrset, or an attset with the fields outPath and ( if required ) bin ( normalized, see example below ), or in its simplest form a store path ( string, outPath of a prepared module ).

NOTE: outPath will be copied/linked “as is”, so do any building/node-gyp stuff first; in particular ensure that bin scripts have proper permissions and are patched using patch-shebangs or patchShebangs.

It is recommended that you use this routine using input from libtree which is designed specifically for this purpose; but you are free to hack together any tree you’d like with various types of inputs.

A single leading node_modules/ path is ignored when creating the tree to make passing from a package-lock.json(v2/3) easy; if you omit this prefix you must do it for all entries because othewise we have to do a ton of conditional regex on every path.

Use node_modules/foo/node_modules/bar for everything, or use foo/node_modules/bar for everything; do not mix them!

addCmd and addBinCmd Arguments

These arguments are functions with the prototype from: to: <STRING> which produce shell code used to add modules/bins to the $node_modules_path/${to}/ directory, taken from the from directory or possibly paths ( in the case of bins ).

You can create your own routines to override these to have them do whatever you want. A notable usage for these which I hope to implement in the future is just to call an arbitrary shell function that can be set in a setup hook; but that is currently left as an exercise for the reader.

Example

An example tree with several types of PKG values:

tree = {
  # Just a store path.
  "node_modules/foo" = "/nix/store/XXXX...-foo";
  # `fetchTree' output has an `outPath' field so this works too
  "node_modules/@bar/bar-core" = builtins.fetchTree { ... };

  # Nested deps
  "node_modules/@bar/bar-core/node_modules/@bar/bar-utils" =
    builtins.fetchTree { ... };

  # With bins
  "node_modules/@blub/quux" = {
    outPath = builtins.fetchTree { ... };
    bin.quux        = "./bin/main.js";
    bin.quux-client = "./client/bin/client.js";
  };

  # local path with "normalized" bindir ( encoded as `__DIR__' ).
  # See `libpkginfo' for more info about `__DIR__'.
  "node_modules/@blub/quux-cli-extras" =
    outPath = ( builtins.path {
      path = ./custom-quux-cli;
      filter = name: type: ( baseNameOf name ) != "node_modules";
    } ).outPath;
    # Points to `./custom-quux-cli/bin'. Pay attention to the quotes.
    bin.__DIR__ = "./bin";
  };

  # From a `pkgEnt' ( see `mkNmDirFromWithSet' as well )
  "node_modules/@my-pkgs/my-ent" = myPkgSet."@my-pkgs/my-ent/0.0.1";

  # A nested dependency yanked from a `pkgSet' with an override.
  "node_modules/bar/node_modules/baz" =
    myPkgSet."baz/1.0.0".prepared.override { runScripts = ["my-script"]; };
};

See additional examples in tests/mkNmDir/tests.nix.

Declaration

Defined in pkgs/mkNmDir/mkNmDirCmd.nix.

# This will read an optional argument `copy = true|false' defaulting
# to symlinks.
# This is to make it less of a headache in the event you want to override a
# symlinked command to be copying.
_mkNmDirCmdWith = {
  tree
# A function taking `from' and `to' as arguments, which should produce a shell
# command that "adds" ( copies/links ) module FROM source directory TO
# module directory.
# See examples above.
# NOTE: Use absolute paths to utilities, this command may be nested as a
# hook in other derivations and you are NOT guaranteed to have `stdenv'
# default path available - not even `coreutils'.
, addCmd ? from: to:
  ( if ( args ? copy ) && ( args.copy == true )
    then _mkNmDirCopyCmd coreutils else _mkNmDirLinkCmd lndir
  ) from to
# Only handle top level `node_modules/.bin` dir.
# This is what you want if  you're only using isolated Nix builders.
# If you're creating an install script for use outside of Nix and you want
# `npm rebuild' and similar commands to work you need those subdirs though.
, ignoreSubBins ? false
# For `package.json' inputs in pure eval mode, we may not know exactly which
# bins need to be linked yet; so we have to perform additional checking and
# globbing at runtime.
# `package-lock.json' inputs don't need to check, and we can skip a lot of IO
# by setting this to false, indicating that your `tree' never contains
# `bin.__DIR__' entries.
# NOTE: We do not process `directories.bin' - you need to normalize your tree
# fields using `libpkginfo' before calling this.
, handleBindir ? true
# Assume every `*/node_modules/*' needs `.bin/' and run `addBinCmd' forall modules. 
# This really exists for use with custom `addBinCmd' routines or incomplete metadata.
# If you use `mkSourceTree' you may find it useful to run `fixup'. 
, assumeHasBin ? false
# Same deal as `addCmd' but for handling bin links.
# This is exposed in case you need to do something wonky like create wrapper
# scripts; but I think it's unlikely that you'll need to.
, addBinCmd ? path: ent:
    if handleBindir then _mkNmDirAddBinCmd       coreutils path ent
                    else _mkNmDirAddBinNoDirsCmd coreutils path ent
# Hooks. You can add arbitrary shell code to run before/after the main routine. 
, preNmDir  ? ""
, postNmDir ? ""
# Input Drvs
, coreutils ? globalArgs.coreutils
, lndir     ? globalArgs.lndir
, ...
} @ args:
# You can't use the `copy' arg AND explicitly set `addCmd'.
assert ( args ? copy ) -> ! ( args ? addCmd ); let ... in ...;

mkNmDirLinkCmd

Calls mkNmDirCmdWith in “link” mode.

Symlinks all modules from Nix store to path indicated by $node_modules_path.

mkNmDirCopyCmd

Calls mkNmDirCmdWith in “copy” mode.

Copies all modules from Nix store to path indicated by $node_modules_path.

mkNmDirPlockV3

• Attrs: { nmDirCmd = { cmd, passthru, meta }; nmDirCmds = { devCopy, devLink, prodCopy, prodLink }; }
  • nmDirCmd, and subattrs in nmDirCmds are all return values from a mkNmDirCmd* call.
  • I have only enumerated the fields in the first nmDirCmd member.
• Functor Args: { tree, addBinCmd?, addCmd?, assumeHasBin?, copy?, coreutils?, dev?, handleBindir?, ignoreSubBins?, lndir?, postNmDir?, preNmDir? }

Example Invocations:

• mkNmDirPlockV3 { pkgSet = ...; }
  • Uses prepared modules defined in a `pkgSet’ collection. These are referenced by key using `lib.idealTreePlockV3’.
• mkNmDirPlockV3 { metaSet = ...; }
  • Fetches sources from `<metaEnt>.fetchInfo’ fields. Nothing is “built” or patched.
• mkNmDirPlockV3 { lockDir = ...; } and mkNmDirPlockV3 { plock = ...; flocoFetch = ...; }
  • Fetches sources from `<metaEnt>.fetchInfo’ fields. Nothing is “built” or patched.
  • This option is ideal if you’re trying to modify `package-lock.json’ manually.

Overview

This is the “magic” package-lock.json(v2/3) -> node_modules/ builder. It’s built on top of lower level functions that allow for fine grained control of how the directory tree is built, what inputs are used, etc; but this form is your “grab a node_modules/ dir off the shelf” routine that tries to do the right thing for a package-lock.json(v2/3).

The resulting attrset is a functor, which just means its an attrset that can be called as a function and reference self.

So out of the box it can become a string, or if you check in subattrs you’ll find myNmd.nmDirCmds.{devLink,devCopy,prodLink,prodCopy}.cmd attrs that lazily generate other styles of copy or tree.

Additionally if you treat it as a function passing args meant for mkNmDir* routines, it will change the settings for the default builder.

The default builder is used for the toString magic, and is stashed under myNmd.nmDirCmd for you to reference. Passing args does NOT modify the 4 “common” builders stashed under nmDirCmds so you can rely on those being there, and if you want you can add more.

Example

tests/pkg-set/tests.nix has a usage example but it’s pretty simple. This goofy example script shows different usages. At the top I’ve included a trivial build pipeline; while your own configuration may conflict with some of these exact calls, it’s a good guide.

with ( builtins.getFlake "at-node-nix" ).legacyPackages.${builtins.currentSystem};
let
  metaSet = lib.metaSetFromPlockV3 { lockDir = toString ./.; };
  pkgSet = builtins.mapAttrs ( path: metaEnt: let
    basePkgEnt = ( mkPkgEntSource metaEnt ) // { nmDirCmd = nmd; };
    doBins = runCommandNoCC basePkgEnt.meta.names.prepared {
      src = basePkgEnt.source;
      # The version on `node-js' that we can't `patchShebangs' to use.
      buildInputs = [nodejs-14_x jq];
    } ''
      cp -r -- "$src" "$out";
      cd "$out";
      mkdir -p .bin;
      jq -r '( .bin // {} )|to_entries[]|"ln -sr -- " + .value + "  ../.bin/" + .key'|sh;
      patchShebangs ../.bin;
    '';
   # NOTE: the calls here to "build" and "install" would not really work. 
   # See the `pkgEnt' and `pkgSet' docs for real advice on build pipelines. 
   in if basePkgEnt.meta.hasBuild         then buildPkgEnt   basePkgEnt else
      if basePkgEnt.meta.hasInstallScript then installPkgEnt basePkgEnt else
      if basePkgEnt.meta.hasBin           then doBins                   else
      basePkgEnt
  ) metaSet.__entries;
  nmd = mkNmDirPlockV3 {
    # Packages will be pulled from here when their "key" ( "<IDENT>/<VERSION>" )
    # matches an attribute in the set.
    inherit pkgSet;
    # Default settings. These are wiped out if you pass args again.
    copy = false;  # Symlink
    dev  = true;   # Include dev modules
  };
  installAnyScript = pkgsFor.writeText "install-nm" ''
    # Automatically converts to a string for current settings.
    installDevLink() {
      cat <<'EOF'|bash
        ${nmd}
        installNodeModules;
      EOF
    }
    # Referring to a stashed command
    installDevCopy() {
      cat <<'EOF'|bash
        ${nmd.nmDirCmds.devCopy.cmd}
        installNodeModules;
      EOF
    }
    # Generating a new command
    installProdLink() {
      cat <<'EOF'|bash
        ${nmd { dev = false; postNm = "echo 'Howdy';" }}
        installNodeModules;
      EOF
    }
    installProdCopy() {
      cat <<'EOF'|bash
        ${nmd { dev = false; copy = true; ignoreSubBins = true; }}
        installNodeModules;
      EOF
    }
    # Install modules to the current working directory.
    export node_modules_path="$PWD/node_modules";

    case "$*" in
      --link\ *\ --dev|--dev\ *\ --link)    installDevLink; ;;
      --copy\ *\ --dev|--dev\ *\ --copy)    installDevCopy; ;;
      --link\ *\ --prod|--prod\ *\ --link)  installProdLink; ;;
      --link\ *\ --prod|--prod\ *\ --link)  installProdLink; ;;
    esac
  '';
in installAnyScript

NOTE: The script above is very naive, it would include a large amount of redundant code; but it’s here to illustrate basic usage. If you actually want to create a script like the one above it makes much more sense to make a simple wrapper around nix eval --raw --expr '...'|sh; to dump scripts.