dockerapache-kafka.net-8.0

How to publish messages to Kafka from another container

I'm using visual studio 2022 to create a containerized application. When I try to connect to kafka I get errors like:

%3|1719311780.957|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1719311780.957|ERROR|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 1/1 brokers are down
%3|1719311780.963|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1719311781.957|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1719311781.957|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1719311811.958|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)
%3|1719311811.959|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)%3|1719311780.957|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1719311780.957|ERROR|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 1/1 brokers are down
%3|1719311780.963|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1719311781.957|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1719311781.957|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1719311811.958|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)
%3|1719311811.959|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)

My yml file for kafka and dependencies looks like this:

version: '3'
services:
  schema-registry:
    image: confluentinc/cp-schema-registry:${CONFLUENT_VERSION:-latest}
    restart: unless-stopped
    depends_on:
      - kafka
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'PLAINTEXT://kafka:9092'
      SCHEMA_REGISTRY_HOST_NAME: 'schema-registry'
      SCHEMA_REGISTRY_LISTENERS: 'http://0.0.0.0:8085'
      SCHEMA_REGISTRY_LOG4J_ROOT_LOGLEVEL: 'INFO'
  akhq:
    image: tchiotludo/akhq
    restart: unless-stopped
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            docker-kafka-server:
              properties:
                bootstrap.servers: "kafka:9092"
              schema-registry:
                url: "http://schema-registry:8085"
              connect:
                - name: "connect"
                  url: "http://connect:8083"
    ports:
      - 8080:8080
    links:
      - kafka
      - schema-registry
  zookeeper:
    ports:
      - '50000:2181'
    image: zookeeper:latest
  kafka:
    ports:
      - '50001:9092'
      - '50002:9093'
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENERS: 'INTERNAL://:9092'
      KAFKA_ADVERTISED_LISTENERS: 'INTERNAL://:9092'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'INTERNAL:PLAINTEXT'
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: '1'
      KAFKA_CREATE_TOPICS: 'example-topic:1:1'
      KAFKA_ADVERTISED_HOST_NAME: host.docker.internal # change to 172.17.0.1 if running on Ubuntu
    image: 'wurstmeister/kafka:latest'
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
volumes: {}

I write the following connection settings in my VS2022 project:

{
  "Kafka": {
    "BootstrapServers": "127.0.0.1:9092",
    "SchemaRegistryUrl": "http://schema-registry:8085"
  }
}

Previously, other options were tried to specify BootstrapServers (localhost:9092, kafka:9092, localhost:50001, kafka:50001), but the errors remain the same.

Please tell me what is my mistake?

Solution

  • This solution was to add to the project file in the PropertyGroup

    <DockerfileRunArguments>--network NetworkNameWhereKafkaRunnedWithDependencies</DockerfileRunArguments>