[SOLVED] mariadb pods replicas crashes

In my Kubernetes cluster, I tried to start a deployment of MariaDB with 3 replicas. After 30 s 2/3 replicas crash, restart, and crashes again. It has something to do with the PVC which is linked to the Deployment. There are some indications that the pods need an exclusive look at some of the files in the PV. One of the pods works as expected.

Similar setups work for both influxDB and Grafana. But something is happening with the MariaDB.

The .yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb
  labels:
    app: mariadb
spec: 
  replicas: 3
  selector:
    matchLabels:
      app: mariadb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - name: mariadb
        image: mariadb:latest
        ports:
        - containerPort: 3306 
        env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: MYSQL_ROOT_PASSWORD
                  name: platform-secret
              [more variables]
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mariadb-pv
      restartPolicy: Always
      volumes:
      - name: mariadb-pv
        persistentVolumeClaim:
          claimName: mariadb-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mariadb-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mariadb
  name: mariadb
spec:
  selector:
    app: influxdb
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
status:
  loadBalancer: {}

I tried to divide the PVC and added to volumes for subfolders, which worked fine:

volumeMounts:
- name: mariadb-aria-storage
  mountPath: /var/lib/mysql/aria
- name: mariadb-innodb-storage
  mountPath: /var/lib/mysql/innodb

So something is wrong with the PVC when mapping /var/lib/mysql

Some error indicators:

kubectl get pods

NAME                       READY   STATUS    RESTARTS      AGE
mariadb-78d7945894-gsjnq   1/1     Running   2 (28s ago)   103s
mariadb-78d7945894-qzfqd   1/1     Running   0             103s
mariadb-78d7945894-zmg7k   1/1     Running   2 (27s ago)   103s

kubectl get event

[...]
2m10s       Warning   FailedAttachVolume       pod/mariadb-78d7945894-zmg7k        AttachVolume.Attach failed for volume "pvc-c5b09340-5060-4b77-921a-9f889bc81e84" : rpc error: code = Internal desc = [ControllerPublishVolume] Attach Volume failed with error failed to attach e162b4b1-be0f-4b63-a8eb-6bf679c9719e volume to cfb3d471-5184-4a79-8474-dc62c4893c7d compute: Bad request with: [POST https://ops.elastx.cloud:8774/v2.1/servers/cfb3d471-5184-4a79-8474-dc62c4893c7d/os-volume_attachments], error message: {"badRequest": {"code": 400, "message": "Invalid input received: Invalid volume: Volume e162b4b1-be0f-4b63-a8eb-6bf679c9719e status must be available or downloading to reserve, but the current status is creating. (HTTP 400) (Request-ID: req-6f2d38f9-c0d4-43ab-81bf-da75d347e723)"}}
2m17s       Warning   FailedScheduling         pod/mariadb-78d7945894-zmg7k        running PreBind plugin "VolumeBinding": Operation cannot be fulfilled on persistentvolumeclaims "mariadb-pvc": the object has been modified; please apply your changes to the latest version and try again
1s          Warning   BackOff                  pod/mariadb-78d7945894-zmg7k        Back-off restarting failed container mariadb in pod mariadb-78d7945894-zmg7k_default(d0952758-94e9-4d9e-aeda-c8f851a891b6)
[...]

kubectl logs mariadb-78d7945894-zmg7k

2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server latest started.
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server latest started.
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2023-11-30 13:31:58 0 [Note] Starting MariaDB 11.2.2-MariaDB-1:11.2.2+maria~ubu2204 source revision 929532a9426d085111c24c63de9c23cc54382259 as process 1
2023-11-30 13:31:58 0 [ERROR] mariadbd: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds
2023-11-30 13:32:28 0 [ERROR] mariadbd: Got error 'Could not get an exclusive lock; file is probably in use by another process' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2023-11-30 13:32:28 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2023-11-30 13:32:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-11-30 13:32:28 0 [Note] InnoDB: Number of transaction pools: 1
2023-11-30 13:32:28 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-11-30 13:32:28 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-11-30 13:32:28 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-11-30 13:32:28 0 [Note] InnoDB: Completed initialization of buffer pool
2023-11-30 13:32:28 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2023-11-30 13:32:28 0 [Note] InnoDB: Check that you do not already have another mariadbd process using the same InnoDB data or log files.
2023-11-30 13:32:28 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-11-30 13:32:28 0 [Note] InnoDB: Starting shutdown...
2023-11-30 13:32:28 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-11-30 13:32:28 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-11-30 13:32:28 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-11-30 13:32:28 0 [ERROR] Could not open mysql.plugin table: "Unknown storage engine 'Aria'". Some plugins may be not loaded
2023-11-30 13:32:28 0 [ERROR] Failed to initialize plugins.
2023-11-30 13:32:28 0 [ERROR] Aborting

If you have any idea, I would be really thankful. If additional information is needed, please ask me!