Hi I have 4 DBs and I need to connect all of them with Debezium Connector. I'm going to float them on AWS EKS and I'm also going to have S3 Sink Connectors, which I expect to be more than 10, divided by schema. What I'm wondering is how many Connect Clusters should I spin up and on what basis should I divide and assign the different connectors to Connect Clusters?
I feel like if I assign them all to one Connect Cluster, there will be some resource or management issues, is that correct? If this is the case, please advise me how to separate the connect clusters and how to distribute the connectors.
Thank you.
I haven't tried it yet and am planning how to operate it.
One kafka connect cluster is able to distribute tasks on its own. There's no specific reason to create unique clusters unless you have monitoring in place and detect resource contention or must use unique configurations between each