Kubernetes Persisten Volumes

I’m working on setting up a Kubernetes pod that needs to work with a Persistent Volume (PV) and Persistent Volume Claim (PVC) for data storage. The goal is to copy a SQLite database file into a secondary directory within the pod, but I’m running into an issue where the file is not found. Here’s the error message I get when I check the logs of the pod: I'm getting an error kubectl logs -f pathfinder-db-snapshot cp: can't stat '/mnt/data/pathfinder/testnet-sepolia.sqlite': No such file or directory

This is my dockerfile

FROM alpine:3.20

RUN apk update && apk add --no-cache \
    sudo \
    bash \
    unzip

WORKDIR /mnt/data

COPY testnet-sepolia.sqlite /mnt/data/pathfinder/testnet-sepolia.sqlite

CMD ["sh", "-c", "sudo -v && mkdir -p /mnt/data/pathfinder-2 && cp /mnt/data/pathfinder/testnet-sepolia.sqlite /mnt/data/pathfinder-2/testnet-sepolia.sqlite && tail -f /dev/null"]

As you can see, I’m copying a file (testnet-sepolia.sqlite) into /mnt/data/pathfinder and later trying to copy it into another directory (/mnt/data/pathfinder-2) when the pod starts.

Here’s the pod configuration that uses the Docker image:

apiVersion: v1
kind: Pod
metadata:
  name: pathfinder-db-snapshot
spec:
  containers:
    - name: pathfinder-fetcher
      image: test/pathfinder-db-snapshotv2
      volumeMounts:
        - mountPath: /mnt/data/pathfinder
          name: pathfinder-volume
        - mountPath: /mnt/data/pathfinder-2
          name: pathfinder-volume-2
  volumes:
    - name: pathfinder-volume
      persistentVolumeClaim:
        claimName: pathfinder-pvc
    - name: pathfinder-volume-2
      persistentVolumeClaim:
        claimName: pathfinder-pvc-2

This is PV

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pathfinder-pv
spec:
  capacity:
    storage: 16Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: "/mnt/data/pathfinder"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pathfinder-pv-2
spec:
  capacity:
    storage: 16Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: "/mnt/data/pathfinder-2"

And this is PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pathfinder-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 16Gi
  storageClassName: manual
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pathfinder-pvc-2
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 16Gi
  storageClassName: manual

The flow basically starts with kind create cluster and then applying pv, pvc and snashot-fetcher pod.

It seems like the file testnet-sepolia.sqlite is not available inside the pod even though it was specified in the Dockerfile, or there may be an issue with mounting the volumes correctly.

Any guidance on what might be going wrong would be greatly appreciated!

In Unix in general, mounting something over a directory hides whatever was previously in that directory. This applies to Docker bind mounts and Kubernetes volume mounts, too: whatever you're mounting into your Pod or container hides whatever is in the underlying image. There's a case of unininitialized Docker named volumes that apparently acts differently, but it's a mistake to depend on it, since it doesn't work in other environments (like Kubernetes) and it has some corner cases (if the image content changes, it will get hidden by older content in the volume).

So the shortest answer to your question is to delete the first PV/PVC/volume/volumeMount that goes over the image content. Without a volume, the source directory for the cp command will come from the image.