kubernetescri-odocker-registry-mirror

imagePullSecrets has no effect and authentication still required

I have a k8s cluster v1.26.4 and all the nodes have crio v1.25.2 installed.

I have private registry with basic auth that mimics as another private registry to reduce code writing. The registry is a docker distribution solution with proxy-mode for latter. Then i have mirror.conf for cri-o as is:

[[registry]]
prefix = private.registry
location = mirror.registry

When i execute
crictl pull --creds private.registry/image:tag
everything is ok. But when i create a pod:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: pod
  name: pod
  namespace: my-ns
spec:
  containers:
    - image: private.registry/image:tag
      name: pod
  imagePullSecrets:
    - name: registry-secret

and a secret:

apiVersion: v1
data:
  .dockerconfigjson: base64-encoded-auths
kind: Secret
metadata:
  name: registry-secret
  namespace: my-ns
type: kubernetes.io/dockerconfigjson

then i have this error:
Failed to pull image "private.registry/image:tag": rpc error: code = Unknown desc = reading manifest tag in mirror.registry/image: unauthorized: authentication required

If i rewrite the mirror.conf for cri-o when prefix and location are the same then no issues whatsoever. This means that my auths in a secret are most likely correct.

[[registry]]
prefix = mirror.registry
location = mirror.registry
Solution

  • Turned out direct requests through crictl pull were not working as well, so the cri-o could be the isse.

    After researching in i've found couple of similar questions (this and this) about missing credentials when using redirect(prefix)/mirror with cri-o. The reason is credentials are stripped when named endpoint and the actual are not the same. It is done intentionally to avoid credentials leaking and commented right in a library project. Looks like there is only one solution to use global_auth_file option in crio.conf file, which works.