Docker build / docker compose up error: Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized

When building a container (through docker build, docker run or docker-compose) using NVIDIA containers, I get the following error, somewhat randomly. When I usually start building the container, it works the first time. But next few times usually don't work.

------
 > [internal] load metadata for nvcr.io/nvidia/l4t-base:r32.5.0:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: 
  failed to create LLB definition: failed to authorize: 
  rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized

My current workaround is to use a slightly different image temporarily whilst I’m adjusting other parts of the Dockerfile.

You have to docker login nvcr.io

Most of this is documented in the setup docs, unfortunately nvcr.io doesn't have very good SEO or UX, so search results and most attempts don't show this page. This is, confusingly, different to ngc catalog / docker login ngc.nvidia.com.

• Install NGC CLI: (for macOS)
  • Run curl -O https://ngc.nvidia.com/downloads/ngccli_mac.zip && unzip ngccli_mac.zip && chmod u+x ngc
  • mv ngc /usr/local/bin/ngc
• Setup your environment:
  • Create an account and get your API key from https://ngc.nvidia.com/
  • Add the API key to the CLI: ngc config set
  • Login to registry: docker login nvcr.io
• Observe docker CLI output containing correct auth details:

[auth] nvidia/tensorrt:pull,push token for nvcr.io

Tips

Make sure you login to the right container registry. It's in the name of the docker image you are using. e.g. nvcr.io/nvidia/tensorrt:22.01-py3 uses nvcr.io.

Explanation

With the benefit of hindsight, I understood failed to fetch anonymous token: unexpected status: 401 Unauthorized. I was not authenticated (or more correctly, I was authenticated to Docker Hub and NGC Catalog instead of nvcr.io). It was giving me an anonymous token, which hit the rate limit. I needed to create an account to get higher limits.