I'm having custom connector developed. I wanted to add dynamic horizontal scalability. So connector, the object of class public class CustomConnector extends SourceConnector creates dynamic number of tasks by providing proper number of task configurations in method public java.util.List<java.util.Map<String, String>> taskConfigs(int maxTasks).
The issue here is that computing the proper number of task configurations takes around 6 minutes. In that time, kafka-connect framework thinks that sth bad happened with the broker, which connector was assigned to, brokerA, and framework reschedules the connector to different broker, brokerB. The brokerA finishes the job and x tasks is created. Then brokerB finishes the computations and I have twice as much tasks than needed.
The question: is there a possibility to increase timeout in framework to wait more time for the tasks?
PS. I know that obvious solution is to make calculations more performant, but it can't be done for know.
The property is being called CONNECT_SCHEDULED_REBALANCE_MAX_DELAY_MS for docker image and scheduled.rebalance.max.delay.ms as property.