[SOLVED] Keep getting "requested access to the resource is denied" when pulling from gitlab

I have a private gitlab repository on gitlab.com (free tier) with a container registry. I successfully built and pushed a docker image to this registry via gitlab-ci.

But I cannot pull the image from the outside (not gitlab-ci but a local docker installation).

Error: requested access to the resource is denied

The logs are from my attempt to use podman but I get the same issue with docker. Tried it with a project token (read-registry) as well as with my personal user and password and on different machines, both linux and macOS.

user@machine:~$ podman login registry.gitlab.com
Authenticating with existing credentials for registry.gitlab.com
Existing credentials are valid. Already logged in to registry.gitlab.com

Pulling:

user@machine:~$ podman pull --log-level=debug registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/user/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 7              
DEBU[0000] Pulling image registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245 (policy: always) 
DEBU[0000] Looking up image "registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" ... 
DEBU[0000] reference "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" does not resolve to an image ID 
DEBU[0000] Trying "registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" ... 
DEBU[0000] reference "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" does not resolve to an image ID 
DEBU[0000] Trying "registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" ... 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/shortnames.conf" 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245 for registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245 
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" 
Trying to pull registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245...
DEBU[0000] Copying source image //registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245 to destination image [overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" 
DEBU[0000] No credentials matching registry.gitlab.com/user.name/myapp/myapp-server found in /run/user/1000/containers/auth.json 
DEBU[0000] Found credentials for registry.gitlab.com/user.name/myapp/myapp-server in credential helper containers-auth.json in file /home/user/.config/containers/auth.json 
DEBU[0000]  No signature storage configuration found for registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245, using built-in default file:///home/user/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.gitlab.com 
DEBU[0000] GET https://registry.gitlab.com/v2/          
DEBU[0000] Ping https://registry.gitlab.com/v2/ status 401 
DEBU[0000] GET https://gitlab.com/jwt/auth?account=user.name&scope=repository%3Auser.name%2Fmyapp%2Fmyapp-server%3Apull&service=container_registry 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] GET https://registry.gitlab.com/v2/user.name/myapp/myapp-server/manifests/0.5.1245 
DEBU[0001] Detected insufficient_scope error, will retry request with updated scope 
DEBU[0001] GET https://gitlab.com/jwt/auth?account=user.name&scope=repository%3Auser.name%2Fmyapp%2Fmyapp-server%3Apull&scope=repository%3Auser.name%2Fmyapp%2Fmyapp-server%3Apull&service=container_registry 
DEBU[0001] Increasing token expiration to: 60 seconds   
DEBU[0001] GET https://registry.gitlab.com/v2/user.name/myapp/myapp-server/manifests/0.5.1245 
DEBU[0001] Content-Type from manifest GET is "application/json" 
DEBU[0001] Discarding non-primary errors:               
DEBU[0001]   unauthorized: authentication required      
DEBU[0001] Accessing "registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245" failed: reading manifest 0.5.1245 in registry.gitlab.com/user.name/myapp/myapp-server: requested access to the resource is denied 
DEBU[0001] Error pulling candidate registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245: initializing source docker://registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245: reading manifest 0.5.1245 in registry.gitlab.com/user.name/myapp/myapp-server: requested access to the resource is denied 
Error: initializing source docker://registry.gitlab.com/user.name/myapp/myapp-server:0.5.1245: reading manifest 0.5.1245 in registry.gitlab.com/user.name/myapp/myapp-server: requested access to the resource is denied
DEBU[0001] Shutting down engines

The entry in the container registry looks like this: