[SOLVED] Grafana Pod is in Init Error state after adding an existing PVC

Grafana Pod is in Init Error state after adding an existing PVC

Installing grafana using helm charts, the deployment goes well and the grafana ui is up, needed to add an existence persistence volume, ran the below cmd:

helm install grafana grafana/grafana -n prometheus --set persistence.enabled=true --set persistence.existingClaim=grafana-pvc

The init container crashes, with the below logs:

kubectl logs grafana-847b88556f-gjr8b -n prometheus -c init-chown-data                    
chown: /var/lib/grafana: Operation not permitted
chown: /var/lib/grafana: Operation not permitted

On checking the deployment yaml found this section:

initContainers:
      - command:
        - chown
        - -R
        - 472:472
        - /var/lib/grafana
        image: busybox:1.31.1
        imagePullPolicy: IfNotPresent
        name: init-chown-data
        resources: {}
        securityContext:
          runAsNonRoot: false
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/grafana
          name: storage
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 472
        runAsGroup: 472
        runAsUser: 472
      serviceAccount: grafana
      serviceAccountName: grafana

Why is the operation failing though its running with runAsUser: 0 ? and the pvc is having access:ReadWriteMany, any workaround ? Or am I missing something

Thanks !!

Solution

NFS turns on root_squash mode by default which functionally disables uid 0 on clients as a superuser (maps those requests to some other UID/GID, usually 65534). You can disable this in your mount options, or use something other than NFS. I would recommend the latter, NFS is bad.