Adding a source does not seem to add it to the store

[fricklerhandwerk]

When I add a large repo as a source, it has to download for a long time. Using that source in a derivation requires downloading it again, apparently. Is that a user error or is it how the tool works?

[piegamesde]

I think that we try to use the correct Nix prefetch call to avoid this, but it might depend on the exact situation. Can you please show a command which adds a source which reproduces this?

[fricklerhandwerk]

I had it on a humongous GitHub repo, but can't reproduce it with something smaller such as Nixpkgs and don't want to try to download gigabytes again...

[piegamesde]

If you encounter it again or manage to extract the respective command from the shell history, feel free to open again.

[eclairevoyant]

I've run into a similar issue myself with the following pin:

    "google-fonts": {
      "type": "Git",
      "repository": {
        "type": "GitHub",
        "owner": "google",
        "repo": "fonts"
      },
      "branch": "main",
      "revision": "3992c83a1afd00abb72a6c237159f20e419433dc",
      "url": "https://github.com/google/fonts/archive/3992c83a1afd00abb72a6c237159f20e419433dc.tar.gz",
      "hash": "1zcqm82c7a08mkkfhm7py494qa41q8pb7x237hdqas01qw5kzcp8"
    },

Updating the pin seems to take a while on its own - presumably the tarball must be downloaded to get the hash.
First build after updating the pin also takes a while, as it tries to redownload the tarball.
The tarball itself is massive, so the time spent is noticeable.

[fricklerhandwerk]

Could it be that this is because npins relies on nix-prefetch-url, which doesn't use the fetcher cache?

[piegamesde]

Yes indeed we shell out to the nix-prefetch-url and nix-prefetch-git commands (https://github.com/andir/npins/blob/master/src/nix.rs). I am a bit confused that these do not cache the result in the store. Are there better alternatives?

[fricklerhandwerk]

nix-prefetch-url produces a store object but doesn't use the fetcher cache, which means that builtins.fetchurl (which happens in import ./npins) will always get a cache miss and only find that the resulting store object already exists after the source was downloaded.

An alternative would be doing the same thing as flakes and storing the NAR hashes in the lockfile, and then constructing the store path via FOD on import.

[piegamesde]

Sounds like a nix bug to me tbh

[fricklerhandwerk]

I'd say it's a lack of deliberate design. A bug would mean something particular was intended but went wrong by accident. The code doesn't look like that to me.

[Lunaphied]

I would say this is an explicit bug, as the documentation for nix-prefetch-url claims

If you don't know the hash, you would have to download the file first, and then fetchurl would download it again when you build your Nix expression. Since fetchurl uses the same name for the downloaded file as nix-prefetch-url, the redundant download can be avoided.

[eclairevoyant]

Per some testing I did, it seems that nix-prefetch-* does in fact cache, but the store path format is different from builtin fetchers. I see this issue was already fixed before, for a different fetcher (nix-prefetch-git vs fetchGit): f0449d0

So a similar workaround for the remaining fetchers should do the trick for now. (Though I still consider it a borderline nix bug that the default behaviour is to induce a cache miss... especially if the docs say "fetchurl uses the same name for the downloaded file as nix-prefetch-url".)