apache-kafkaapache-kafka-streams

How do I control the names of the internal topics in my kafka streams app for a leftJoin operation?


I'm building a kafka streams app in an environment where the app will not be allowed to have admin privileges to create internal topics. I can get around this limitation by making my operations have predictable names and creating them in advance (though I'll need to be careful about making sure they're configured correctly).

In my app's topology, I've got some state, a couple of selectKey operations, and a leftJoin. I've figured out how to use Named.as() in the repartition operations that come from selectKey and defining the changelog topic names for the state are pretty straightforward, but I'm having some trouble with the internal topics that come from the leftJoin. When I run my app locally with a kafka sidecar (where I do have admin privileges), I can see topics like this:

my-application-id-KSTREAM-JOINTHIS-0000000014-store-changelog
my-application-id-KSTREAM-OUTEROTHER-0000000015-store-changelog

I understand why those topics exist, but my question is this: outside of the my-application-id prefix, do I have any control over the naming of these topics? Is there an API or hook I can use to control the inner portion of the topic name?

(This is a KStream <-> KStream leftJoin if that makes a difference.)


Solution

  • KStream-KStream joins accept a StreamJoined parameter that allows you to set the name. Was added in Apache Kafka 2.4 release (cf https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join)