nix

Why do some Nix expressions not cache properly?

I’m trying to use Nix in a CI scenario to build a local binary cache using mkBinaryCache so that I can do Nix builds in containers, without downloading lots of stuff from cache.nixos.org.

I’m finding that some Nix expressions don’t hit in the cache in the way I expect.

Test setup

  1. Let EXPR be some Nix expression.
  2. Build pkgs.mkBinaryCache { rootPaths = [ (${EXPR}) ]; }, and save the path as BUILT_CACHE
  3. Let SUBSTITUTER="file://${BUILT_CACHE}?trusted=true"
  4. Make a temporary dir to use as a fresh store, and run the following
nix-build -E "$EXPR" -v \
   --store "$STORE" \
   --option substituters "$SUBSTITUTER" \
   --option trusted-substituters "$SUBSTITUTER" \
   --no-link \
   --max-jobs 0

My goal is for this to succeed using only downloads from the substituter. Note the --max-jobs 0, which should keep Nix from doing any local builds.

I have this whole process as a Bash script here.

Results

If EXPR is pkgs.hello, then this works.

If EXPR is pkgs.symlinkJoin { name = "test"; paths = []; }, it does NOT work. It shows long list of derivations it wants to build, seemingly in order to realize some tools like lndir. I guess it wants to built the symlink folder itself.

This sort of goes against my mental model of how Nix works. I thought it would calculate the expected store path for the symlink join expression, and then query the cache to see if it’s already there.

If I examine the binary cache dir, I can see that there is a .narinfo corresponding to the desired output. But AFAICT from looking at the verbose build logs, Nix doesn’t even query the substituter for this path. What’s going on?

I also posted this question to Nix Discourse here.

Solution

  • Aha, I figured it out: I needed the always-allow-substitutes option.

    Just add the --option always-allow-substitutes true flag to the nix-build command.