dockercachingdocker-registrydocker-registry-mirror

Docker-Registry in Pull-Through configuration not working

I'm using the official Docker-registry image to serve as a pull-through cache.

I tried to start it as simple as possible, to minimize sources of error.

$ docker run --rm \
    -p 5000:5000 \
    --env REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io" \
    --env REGISTRY_LOG_LEVEL=debug \
    registry

Only the proxy option is set, to make the cache pull-through. However pulling through doesn't seem to want to work in any configuration I have tried. Startup generates the expected Log output:

time="2021-08-13T11:57:33.0779466Z" level=info msg="Starting upload purge in 15m0s" go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry version=v2.7.1 
time="2021-08-13T11:57:33.0778389Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry version=v2.7.1 
time="2021-08-13T11:57:33.0780354Z" level=info msg="redis not configured" go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry version=v2.7.1 
time="2021-08-13T11:57:33.0842915Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry version=v2.7.1
time="2021-08-13T11:57:33.0844169Z" level=debug msg="filesystem.Stat("/scheduler-state.json")" go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry trace.duration=35.6µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=da97b9e2-9326-4fef-815e-d50b5f61d92c trace.line=155 version=v2.7.1 
time="2021-08-13T11:57:33.0844337Z" level=info msg="Starting cached object TTL expiration scheduler..." go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry version=v2.7.1 
time="2021-08-13T11:57:33.6404953Z" level=info msg="Discovered token authentication URL: https://auth.docker.io/token" go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c 
time="2021-08-13T11:57:33.6407066Z" level=info msg="Registry configured as a proxy cache to https://registry-1.docker.io" go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry version=v2.7.1 
time="2021-08-13T11:57:33.6407596Z" level=warning msg="Registry does not implement RempositoryRemover. Will not be able to delete repos and tags" go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry version=v2.7.1 
time="2021-08-13T11:57:33.6420203Z" level=info msg="listening on [::]:5000" go.version=go1.11.2 instance.id=d5d52c10-f696-4b21-af33-68d581124e0c service=registry version=v2.7.1

Now we try to pull an image through the cache:

$ docker pull localhost:5000/hello-world

However this doesn't produce the expected result. it just says:

Using default tag: latest
Error response from daemon: manifest for localhost:5000/hello-world:latest not found: manifest unknown: manifest unknown

And the Log tells me (I do not understand, why there's a 404-error):

172.17.0.1 - - [13/Aug/2021:12:07:53 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \\(windows\\))"
time="2021-08-13T12:07:53.6733433Z" level=debug msg="authorizing request" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=e0715e2f-60cf-47e5-a651-a08e146a0528 http.request.method=GET http.request.remoteaddr="172.17.0.1:34814" http.request.uri="/v2/" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))"
time="2021-08-13T12:07:53.6733875Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=e0715e2f-60cf-47e5-a651-a08e146a0528 http.request.method=GET http.request.remoteaddr="172.17.0.1:34814" http.request.uri="/v2/" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration="573.5µs" http.response.status=200 http.response.written=2
time="2021-08-13T12:07:53.6768547Z" level=debug msg="authorizing request" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=6e6e3439-9079-4c53-aa2b-6496447e9291 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:34818" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" vars.name=hello-world vars.reference=latest   
time="2021-08-13T12:07:53.6773942Z" level=debug msg=GetImageManifest go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=6e6e3439-9079-4c53-aa2b-6496447e9291 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:34818" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" vars.name=hello-world vars.reference=latest        
time="2021-08-13T12:07:54.166909Z" level=info msg="Challenge established with upstream : {https   registry-1.docker.io /v2/  %!s(bool=false)  } &{{{%!s(int32=0) %!s(uint32=0)} %!s(uint32=0) %!s(uint32=0) %!s(int32=0) %!s(int32=0)} map[https://registry-1.docker.io:443/v2/:[{bearer map[realm:https://auth.docker.io/token service:registry.docker.io]}]]}" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=6e6e3439-9079-4c53-aa2b-6496447e9291 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:34818" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" vars.name=hello-world vars.reference=latest
time="2021-08-13T12:07:55.2663453Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/hello-world/_manifests/tags/latest/current/link")" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=6e6e3439-9079-4c53-aa2b-6496447e9291 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:34818" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" trace.duration=47.2µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=5dfea3e4-682c-433e-854f-aca90bee9454 trace.line=95 vars.name=hello-world vars.reference=latest
time="2021-08-13T12:07:55.266465Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=latest" err.message="manifest unknown" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=6e6e3439-9079-4c53-aa2b-6496447e9291 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:34818" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.5914358s http.response.status=404 http.response.written=96 vars.name=hello-world vars.reference=latest
172.17.0.1 - - [13/Aug/2021:12:07:53 +0000] "HEAD /v2/hello-world/manifests/latest HTTP/1.1" 404 96 "" "docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \\(windows\\))"
time="2021-08-13T12:07:55.2684757Z" level=debug msg="authorizing request" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=e0810233-c118-483c-a308-09462f8e0923 http.request.method=GET http.request.remoteaddr="172.17.0.1:34828" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" vars.name=hello-world vars.reference=latest    
time="2021-08-13T12:07:55.2691789Z" level=debug msg=GetImageManifest go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=e0810233-c118-483c-a308-09462f8e0923 http.request.method=GET http.request.remoteaddr="172.17.0.1:34828" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" vars.name=hello-world vars.reference=latest
time="2021-08-13T12:07:55.5995189Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/hello-world/_manifests/tags/latest/current/link")" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=e0810233-c118-483c-a308-09462f8e0923 http.request.method=GET http.request.remoteaddr="172.17.0.1:34828" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" trace.duration=102.8µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=57fc368b-5205-457d-b78d-e7871bedd661 trace.line=95 vars.name=hello-world vars.reference=latest
time="2021-08-13T12:07:55.5999262Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=latest" err.message="manifest unknown" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=e0810233-c118-483c-a308-09462f8e0923 http.request.method=GET http.request.remoteaddr="172.17.0.1:34828" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=332.3265ms http.response.status=404 http.response.written=96 vars.name=hello-world vars.reference=latest
172.17.0.1 - - [13/Aug/2021:12:07:55 +0000] "GET /v2/hello-world/manifests/latest HTTP/1.1" 404 96 "" "docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \\(windows\\))"
time="2021-08-13T12:07:57.6108264Z" level=debug msg="filesystem.Stat("/")" go.version=go1.11.2 instance.id=dd26b9f4-62d7-4df3-ab3f-fab82ad90e21 service=registry trace.duration=30.8µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=5eff62d8-0d64-4369-b728-d7cda8879971 trace.line=155 version=v2.7.1
time="2021-08-13T12:08:07.6115141Z" level=debug msg="filesystem.Stat("/")" go.version=go1.11.2 instance.id=dd26b9f4-62d7-4df3-ab3f-fab82ad90e21 service=registry trace.duration=78.5µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=6ee865b2-2d0f-4f6e-b564-e3f5e7a333c2 trace.line=155 version=v2.7.1

I think this line is the important one:

level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=latest" err.message="manifest unknown" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=6e6e3439-9079-4c53-aa2b-6496447e9291 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:34818" http.request.uri="/v2/hello-world/manifests/latest" http.request.useragent="docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \(windows\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.5914358s http.response.status=404 http.response.written=96 vars.name=hello-world vars.reference=latest

This is the windows-wsl log output. However the same exact configuration performed (or didn't perform) the same on a native ubuntu 20.4 machine.

Solution

  • You need to add your new registry to daemon.json configuration file (Docs).

    {
  // other options ...

  "registry-mirrors": [
    "http://localhost:5000"
  ]
}

    And then pull images by using docker pull without registry address: docker pull alpine. It will cache the image inside a registry container (you can mount folder to persist the cache between restarts).