mongodbdockerapache-kafkaapache-kafka-connectdebezium

Connection Timeout: Cant connect mongo with debezium for CDC


I have mongo and kafka connect set up in docker-compose. I want to enable CDC from mongo with debezium. These are my services:

  mongodb:
    image: mongo
    restart: always
    command: mongod --replSet debezium --bind_ip_all
    environment:
      MONGO_INITDB_DATABASE: test
      MONGO_INITDB_FEATURES: "1"
    expose:
      - "27017"
    ports:
      - "27017:27017"
    volumes:
      - ~/mongos:/data/db

  debezium:
    image: debezium/connect:latest
    restart: always
    container_name: debezium
    hostname: debezium
    depends_on:
      - postgres
      - kafka
      - mongodb
    ports:
      - '8083:8083'
    environment:
      BOOTSTRAP_SERVERS: kafka:29092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: connect_configs
      STATUS_STORAGE_TOPIC: connect_statuses
      OFFSET_STORAGE_TOPIC: connect_offsets
      KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      ENABLE_DEBEZIUM_SCRIPTING: 'true'
    healthcheck:
      test:
        [
          'CMD',
          'curl',
          '--silent',
          '--fail',
          '-X',
          'GET',
          'http://localhost:8083/connectors',
        ]
      start_period: 10s
      interval: 10s
      timeout: 5s
      retries: 5

All the other relevant services are working as well since I am already using postgres with debezium. In case of mongo, the connector always returns "Connection Refused". I created a replica set in the mongodb and initialized a user with root privilege:

Initializing Replica Set rs.initiate({_id: "debezium", members:[{_id: 0, host: "localhost:27017"}]}) Creating User with Permission use admin db.createUser({ user: "debezium", pwd: "dbz", roles: ["dbOwner"] })

This is my connector.json:

{
  "name": "mongo-db-connector",
  "config": {
    "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
    "tasks.max": "1",
    "topic.prefix": "dbserver2",
    "mongodb.connection.string": "mongodb://debezium:dbz@mongodb:27017/?replicaSet=debezium",
    "database.include.list": "test",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.drop.tombstones": "true"
  }
}

I am able to connect with mongo replica set using the mongodb compass running locally; however, when i run the connector it always returns me, Unable to connect: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}.

I have double checked if the replica set has been initialized correctly as well as the user permissions. I can also create a postgres connector. I am assuming its not a network issue since all of this is in a docker-compose.


Solution

  • Kudos to @OneCricketeer. I have now defined hostname: mongodb within my MongoDB service. Previously, I was also incorrectly defining the replica set: It was rs.initiate({_id: "debezium", members:[{_id: 0, host: "localhost:27017"}]}); I changed it to rs.initiate({_id: "debezium", members:[{_id: 0, host: "mongodb:27017"}]}) and now it works. For testing purposes, I also removed the debezium:dbz (username:password) in the connection string.