We have two URLs for our registry: one internal in our VPC and one external for customers pulling our images. We switched to a digest based reference system, so we pull images by their sha256 digests. Now we also want to give customers the option to install without internet access, so we export the images using docker save and then can load them using docker load. Unfortunately, we are having issues persisting the digest in this process.
How can I transfer a digest from one registry name to another when using docker tag?
An example might be illuminating here:
$ docker pull internal.registry.local/development/img:0.23.0@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267: Pulling from development/img
Digest: sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
Status: Image is up to date for internal.registry.local/development/img:0.23.0@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
internal.registry.local/development/img:0.23.0@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
gives me my image under the dev registry
$ docker image ls --digests
internal.registry.local/development/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
and is equipped with the digest
$ docker image inspect internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 | jq '.[] | { Id, RepoTags, RepoDigests }'
{
"Id": "sha256:dc9c4901ced19676f90c95d0f82c85ba97d15ba1c39d38ca9d692f3d3658bd43",
"RepoTags": [
"inernal.registry.local/development/img:0.23.0"
],
"RepoDigests": [
"internal.registry.local@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
]
}
If I now add another tag to that image under the same registry, the digest is transferred
$ docker tag internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 internal.registry.local/development/img:0.23.0-custom
$ docker image ls --digests
internal.registry.local/development/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
internal.registry.local/development/img 0.23.0-custom sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
$ docker image inspect internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 | jq '.[] | { Id, RepoTags, RepoDigests }'
{
"Id": "sha256:dc9c4901ced19676f90c95d0f82c85ba97d15ba1c39d38ca9d692f3d3658bd43",
"RepoTags": [
"internal.registry.local/development/img:0.23.0",
"internal.registry.local/development/img:0.23.0-custom",
],
"RepoDigests": [
"internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
]
}
But if I change the registry, the digest is lost
$ docker tag internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 external.example.org/production/img:0.23.0
$ docker image ls --digests
internal.registry.local/development/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
internal.registry.local/development/img 0.23.0-custom sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
external.example.org/production/img 0.23.0 <none> dc9c4901ced1 8 weeks ago 10.7GB
$ docker image inspect internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 | jq '.[] | { Id, RepoTags, RepoDigests }'
{
"Id": "sha256:dc9c4901ced19676f90c95d0f82c85ba97d15ba1c39d38ca9d692f3d3658bd43",
"RepoTags": [
"internal.registry.local/development/img:0.23.0",
"internal.registry.local/development/img:0.23.0-custom",
"external.example.org/production/img:0.23.0"
],
"RepoDigests": [
"internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
]
}
But mind you, the registry is actually the same, just under a different host name. So I can pull the image by digest from the external name and get the digest attached to it
$ docker pull external.example.org/production/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
external.example.org/production/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267: Pulling from production/img
Digest: sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
Status: Downloaded newer image for external.example.org/production/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
external.example.org/production/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267
$ docker image ls --digests
internal.registry.local/development/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
internal.registry.local/development/img 0.23.0-custom sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
external.example.org/production/img 0.23.0 sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 dc9c4901ced1 8 weeks ago 10.7GB
$ docker image inspect internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267 | jq '.[] | { Id, RepoTags, RepoDigests }'
{
"Id": "sha256:dc9c4901ced19676f90c95d0f82c85ba97d15ba1c39d38ca9d692f3d3658bd43",
"RepoTags": [
"internal.registry.local/development/img:0.23.0",
"internal.registry.local/development/img:0.23.0-custom",
"external.example.org/production/img:0.23.0"
],
"RepoDigests": [
"internal.registry.local/development/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
"external.example.org/production/img@sha256:9c3c425cc0114e358c58800b544e104be5d5c8f3b594871dafbaf9f28444d267"
]
}
How can I get the digest attached to the imgage when changing the registry by re-tagging without pulling by digest from the new registry name?
I've tried all permutations of
docker tag A/img@sha B/img:tag
docker tag A/img:tag B/img:tag
docker tag A/img:tag@sha B/img:tag
docker tag A/img@sha B/img@sha # error
no no avail.
The digest is made from json that is specific to how the image was pushed to the registry (different tools could change the whitespace or ordering of fields in the json to get a different sha256 hash). So docker doesn't know this value until you push or pull from the registry.
When you try to do a save/load, you'll see this issue again because the saved docker format has it's own manifest that doesn't have the same json format of what's pushed to the registry, so the digest will be regenerated when pushing to a registry after the docker load is run.
You can ship the images offline as a tar using the OCI Layout, which includes the original manifest, byte for byte, to avoid changing the digest. However, this won't load directly into the docker engine. The remote site will need to first import the images into their own registry. There are a few tools for importing and exporting between a registry and the OCI layout, including crane (from Google's go-containerregistry), skopeo (from RedHat), and regclient (from myself).
Using regclient's regctl command, that looks like:
regctl image export registry.example.org/repo@sha256... file.tar
# ... transport tar file to remote location ...
regctl image import client.example.com/repo@sha256... file.tar
Note, you typically still want to tag the image on the registry to avoid garbage collection of the untagged manifest. This won't affect the ability to pull by digest.