RabbitMQ fails to start with persistence storage on Kubernetes. Permission denied error

I have a rabbit mq pod and I configured to use a persistence storage incase of pod restart/deletion by mounting a volume.

I configured everything but not able to get through this error:

/usr/lib/rabbitmq/bin/rabbitmq-server: 42: 
/usr/lib/rabbitmq/bin/rabbitmq-server: 
cannot create /var/lib/rabbitmq/mnesia/rabbit@reana-message-broker-5f45f797ff-cs79m.pid: 
Permission denied

Here're my config file and deployment app for kubernetes

1. Dockerfile

FROM ubuntu:16.04
# hadolint ignore=DL3009
RUN apt-get update
# hadolint ignore=DL3008
RUN apt-get -y install --no-install-recommends rabbitmq-server
RUN apt-get -y autoremove && apt-get -y clean
# hadolint ignore=DL3001
RUN service rabbitmq-server start
COPY start.sh /start.sh
RUN chmod 755 ./start.sh
EXPOSE 5672
EXPOSE 15672
CMD ["/start.sh", "test", "1234"]

2. start.sh

#!/bin/sh
cat > /etc/rabbitmq/rabbitmq.conf <<EOF
listeners.tcp.default = 5672

default_user = <<"$1">>
default_pass = <<"$2">>
EOF
rabbitmq-server

3. rabbitmq.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: message-broker
  namespace: {{ .Release.Namespace }}
spec:
  ports:
   - port: 5672
     targetPort: 5672
     name: "tcp"
     protocol: TCP
   - port: 15672
     targetPort: 15672
     name: "management"
     protocol: TCP
  selector:
    app: message-broker
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: message-broker
  namespace: {{ .Release.Namespace }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: message-broker
  template:
    metadata:
      labels:
        app: message-broker
    spec:
      containers:
      - name: message-broker
        image: {{ .Values.message_broker.image }}
        imagePullPolicy: {{ .Values.components.message_broker.imagePullPolicy }}
        ports:
        - containerPort: 5672
          name: tcp
        - containerPort: 15672
          name: management
        volumeMounts:
        - name: data
          mountPath: /var/lib/rabbitmq/mnesia
        env:
        - name: RABBITMQ_DEFAULT_PASS
          valueFrom:
            secretKeyRef:
              name: rabbitmq-secrets
              key: password # password = root
        - name: RABBITMQ_DEFAULT_USER
          valueFrom:
            secretKeyRef:
              name: rabbitmq-secrets
              key: user # user = root
      ...
      nodeSelector:
      ....
      volumes:
      - name: data
        hostPath:
          path: /var/test/rabbitmq

Let me know what I might be missing. :)

The volume you mounted in /var/lib/rabbitmq/mnesia is owned by root.

The rabbitmq process is running as rabbitmq user and doesn't have write access to this directory.

In your start.sh add:

chown rabbitmq:rabbitmq /var/lib/rabbitmq/mnesia

before starting the rabbitmq-server process.