Producer serializes the message and send them to Broker in byte arrays. And Consumers deserializes those byte arrays. Broker always stores and passes byte arrays. This is how I understood.
But when you use REST Proxy in Kafka, Producer encodes the message with base64, and Consumer decodes those base64 messages.
A Python example of Producer and Consumer :
# Producer using the REST Proxy
payload = {"records" :
[{
"key":base64.b64encode("firstkey"),
"value":base64.b64encode("firstvalue")
}]}
# Consumer using the REST Proxy
print "Message Key:" + base64.b64decode(message["key"])
Why do you send message in base64 to the Broker instead of byte arrays? When using REST Proxy, a Broker stores messages in base64 format?
When a Producer wants to send a message 'Man', it serializes into bytes (bits). A Broker will store it as 010011010110000101101110
. When a Consumer gets this message, it will deserialize back to Man.
However, according to Confluent document :
Data formats - The REST Proxy can read and write data using JSON, raw bytes encoded with base64 or using JSON-encoded Avro.
Therefore, a Producer using REST Proxy will change the message Man
into TWFu
(base64 encode) and send this to a Broker, and a Consumer using REST Proxy will base64 decode this back to Man
.