I run an IoT platform that relies on MQTT (my broker is aedes) for some features. To scale my backend better, I started implementing a Kafka service. The devices will speak with MQTT, which will relay messages to Kafka, and the backend will subscribe to Kafka instead of MQTT.
However, according to Kafka's documentation :
When suppling a regular expression, the consumer will not match topics created after the subscription. If your broker has topic-A and topic-B, you subscribe to /topic-.*/, then topic-C is created, your consumer would not be automatically subscribed to topic-C.
This is totally different from MQTT, with whom a client can subscribe to wildcard before the topics are created.
Since my topic contains device-<device-id>-stuff, I need to create the re-subscribe every time a new device was created. Also, I need to create the topics manually from the backend, instead of waiting for my device to publish the first message on it. And I become highly dependent on the "existing topic list" persisted somewhere, if I start my cluster from scratch I will have recreate all topics again.
How to avoid all this tricky patch ? Did I miss something from the documentation ?
You're reading KafkaJS documentation, rather than Kafka itself. The JVM consumer client will refresh its metadata every 5 minutes, by default, to find new topics/partitions matching the subscription pattern.
If you need the same in kafkajs, you may need to file an issue with that repo.
Or maybe other options like node-rdkafka work differently
become highly dependent on the "existing topic list" persisted somewhere
It's already stored in Zookeeper