I am running a kafka cluster and I wanted to migrate to kafka 4.0 however I am not sure if all connected producer and consumers are compatible with it. I've already looked at kafka-consumer-list thread.
Is there a way to figure out if they are compatible (or at least what versions do they run) without going into each individual repository and manually checking?
Is the client version even sent to the cluster in metadata?
update 02.04.2025
I've found some documentation regarding this topic however it goes in my case in wrong directions, because i need to ask from broken, clients versions.
In order to work against multiple broker versions, clients need to know what versions of various APIs a broker supports. The broker exposes this information since 0.10.0.0...
The REST Consumer API gives a peek into
the current state of consumer groups, allows you to create a consumer in a consumer group and consume messages from topics and partitions
From the server side the best way to check if your clients are compatible with Kafka 4.0 is to use metrics.
First start with the message format conversion to ensure all your clients are using message version v2 (this is mostly for sanity as v2 has been the default since Kafka 0.11). To do so you can check kafka.network:type=RequestMetrics,name=MessageConversionsTimeMs,request={Produce|Fetch}. If all clients are using message format v2, this should be 0.
Clients can provide their name and version when connecting. You can see if you have any unknown clients using kafka.server:type=socket-server-metrics,clientSoftwareName=unknown,clientSoftwareVersion=unknown,listener=.*,networkProcessor=.*. If the connections value is not zero don't panic, several third party clients still don't provide their names.
If you have unknown clients, you can check the APIs they use with kafka.network:type=RequestMetrics,name=RequestsPerSec,request=.*,version=.*. The minimum version for each API that Kafka 4.0 supports are listed in https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0