Error Starting Multiple Kafka Brokers with KRaft using Docker Compose: Metadata and Checkpoint File Issues

When attempting to deploy multiple Kafka brokers using Docker Compose with KRaft consensus protocol, the setup fails to initialize correctly. Each Kafka broker is configured to use KRaft (KAFKA_CFG_PROCESS_ROLES=controller,broker), with listeners and advertised listeners defined for each node (kafka-1, kafka-2, kafka-3). Despite configuring these settings, the brokers encounter errors related to metadata file creation (Error while writing meta.properties file) during startup. These errors prevent the brokers (kafka-2 and kafka-3) from fully initializing and cause them to exit with code 1. The setup works when deploying a single broker (kafka-1), suggesting that the issue specifically pertains to configuring multiple brokers with KRaft using Docker Compose.

yml

version: '3.8'

services:
  kafka-1:
    image: bitnami/kafka:latest
    user: 'root'
    ports: 
      - "9092:9092"
    environment:
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_KRAFT_CLUSTER_ID=WnLkTHhk
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9091,2@kafka-2:9091,3@kafka-3:9091
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9090,CONTROLLER://:9091,EXTERNAL://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-1:9090,EXTERNAL://${HOSTNAME:-localhost}:9092
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENERS_NAME=PLAINTEXT
    volumes:
      - kafka-data:/bitnami/kafka

  kafka-2:
    image: bitnami/kafka:latest
    user: 'root'
    ports: 
      - "9094:9094"
    environment:
      - KAFKA_CFG_NODE_ID=2
      - KAFKA_KRAFT_CLUSTER_ID=WnLkTHhk
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9091,2@kafka-2:9091,3@kafka-3:9091
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9090,CONTROLLER://:9091,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-2:9090,EXTERNAL://${HOSTNAME:-localhost}:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENERS_NAME=PLAINTEXT
    volumes:
      - kafka-data:/bitnami/kafka

  kafka-3:
    image: bitnami/kafka:latest
    user: 'root'
    ports: 
      - "9096:9096"
    environment:
      - KAFKA_CFG_NODE_ID=3
      - KAFKA_KRAFT_CLUSTER_ID=WnLkTHhk
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9091,2@kafka-2:9091,3@kafka-3:9091
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9090,CONTROLLER://:9091,EXTERNAL://:9096
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-3:9090,EXTERNAL://${HOSTNAME:-localhost}:9096
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENERS_NAME=PLAINTEXT
    volumes:
      - kafka-data:/bitnami/kafka

volumes:
  kafka-data:

env file:

HOSTNAME=host.docker.internal

ERROR:

kafka-1_1  | kafka 22:33:12.35 INFO  ==> Formatting storage directories to add metadata...
kafka-3_1  | metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty, dirs={/bitnami/kafka/data: EMPTY})
kafka-3_1  | Formatting /bitnami/kafka/data with metadata.version 3.7-IV4.
kafka-2_1  | metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty, dirs={/bitnami/kafka/data: EMPTY})
kafka-2_1  | Formatting /bitnami/kafka/data with metadata.version 3.7-IV4.
kafka-1_1  | metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty, dirs={/bitnami/kafka/data: EMPTY})
kafka-1_1  | Formatting /bitnami/kafka/data with metadata.version 3.7-IV4.
kafka-3_1  | Error while writing meta.properties file /bitnami/kafka/data: /bitnami/kafka/data/bootstrap.checkpoint.tmp
kafka-2_1  | Error while writing meta.properties file /bitnami/kafka/data: /bitnami/kafka/data/bootstrap.checkpoint.tmp
kafka-1_1  | kafka 22:33:14.22 INFO  ==> ** Kafka setup finished! **
kafka-1_1  | 
kafka-1_1  | kafka 22:33:14.24 INFO  ==> ** Starting Kafka **
architecture-experiments_kafka-2_1 exited with code 1
architecture-experiments_kafka-3_1 exited with code 1

when executing docker-compose --env-file environment.env up

Configuration Adjustment, with unique KAFKA_CFG_NODE_ID, KAFKA_CFG_PROCESS_ROLES, and KAFKA_CFG_CONTROLLER_QUORUM_VOTERS environment variables for each broker. Successfully configured a single Kafka container (kafka-1) with appropriate IP definitions using /etc/hosts within Docker Compose

i had the same problem, the error is that you mount the same volum for two brokers.