I have been trying to use kafkacat to find a message in a topic and publish it back into the topic. We use protobuf so the message values should be in bytes (Keys can be different such as strings or bytes). However, I am unable to publish the message that could be deserialized properly.
How can I do this with kafkacat? I am also open to using other recommended tools for doing this.
Example attempt:
kafkacat -b <broker> -t <topic> -o -10 -e -c 1 -C -K: > test2.txt
cat test2.txt | kafkacat -b <broker> -t <topic> -P -K:
test2.txt shows:
21aa7e2f-41a1-4972-9108-3057627d53f0:
Y/<protobuf.class.path>i
$21aa7e2f-41a1-4972-9108-3057627d53f0A
DIABETEBG_METERBG300"BG300*QP3687WK02000012????
But when I use the same kafkacat consumer command to get the result, I just get the last line:
DIABETEBG_METERBG300"BG300*QP3687WK02000012????
I think the problem is that the consuming outputs lines (might be a part of the payload?) And the producer is treating each line as a new message.
the producer is treating each line as a new message
That's correct.
If you have a binary file, I suggest writing code for this, as kafkacat assumes UTF8 encoded strings as input