I have mongo and kafka connect set up in docker-compose. I want to enable CDC from mongo with debezium. These are my services:
mongodb:
image: mongo
restart: always
command: mongod --replSet debezium --bind_ip_all
environment:
MONGO_INITDB_DATABASE: test
MONGO_INITDB_FEATURES: "1"
expose:
- "27017"
ports:
- "27017:27017"
volumes:
- ~/mongos:/data/db
debezium:
image: debezium/connect:latest
restart: always
container_name: debezium
hostname: debezium
depends_on:
- postgres
- kafka
- mongodb
ports:
- '8083:8083'
environment:
BOOTSTRAP_SERVERS: kafka:29092
GROUP_ID: 1
CONFIG_STORAGE_TOPIC: connect_configs
STATUS_STORAGE_TOPIC: connect_statuses
OFFSET_STORAGE_TOPIC: connect_offsets
KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
ENABLE_DEBEZIUM_SCRIPTING: 'true'
healthcheck:
test:
[
'CMD',
'curl',
'--silent',
'--fail',
'-X',
'GET',
'http://localhost:8083/connectors',
]
start_period: 10s
interval: 10s
timeout: 5s
retries: 5
All the other relevant services are working as well since I am already using postgres with debezium. In case of mongo, the connector always returns "Connection Refused". I created a replica set in the mongodb and initialized a user with root privilege:
Initializing Replica Set rs.initiate({_id: "debezium", members:[{_id: 0, host: "localhost:27017"}]}) Creating User with Permission use admin db.createUser({ user: "debezium", pwd: "dbz", roles: ["dbOwner"] })
This is my connector.json:
{
"name": "mongo-db-connector",
"config": {
"connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
"tasks.max": "1",
"topic.prefix": "dbserver2",
"mongodb.connection.string": "mongodb://debezium:dbz@mongodb:27017/?replicaSet=debezium",
"database.include.list": "test",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "false",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.drop.tombstones": "true"
}
}
I am able to connect with mongo replica set using the mongodb compass running locally; however, when i run the connector it always returns me, Unable to connect: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]\nYou can also find the above list of errors at the endpoint
/connector-plugins/{connectorType}/config/validate"}
.
I have double checked if the replica set has been initialized correctly as well as the user permissions. I can also create a postgres connector. I am assuming its not a network issue since all of this is in a docker-compose.
Kudos to @OneCricketeer. I have now defined hostname: mongodb
within my MongoDB service. Previously, I was also incorrectly defining the replica set:
It was rs.initiate({_id: "debezium", members:[{_id: 0, host: "localhost:27017"}]})
; I changed it to rs.initiate({_id: "debezium", members:[{_id: 0, host: "mongodb:27017"}]}) and now it works. For testing purposes, I also removed the debezium:dbz
(username:password) in the connection string.