apache-kafkakafka-rest

Why base64 encode/decode in Kafka REST Proxy?


Producer serializes the message and send them to Broker in byte arrays. And Consumers deserializes those byte arrays. Broker always stores and passes byte arrays. This is how I understood.

But when you use REST Proxy in Kafka, Producer encodes the message with base64, and Consumer decodes those base64 messages.

A Python example of Producer and Consumer :

# Producer using the REST Proxy

payload = {"records" : 
   [{
        "key":base64.b64encode("firstkey"),
        "value":base64.b64encode("firstvalue")
   }]}
# Consumer using the REST Proxy 

   print "Message Key:" + base64.b64decode(message["key"])

Why do you send message in base64 to the Broker instead of byte arrays? When using REST Proxy, a Broker stores messages in base64 format?


Solution

  • enter image description here

    When a Producer wants to send a message 'Man', it serializes into bytes (bits). A Broker will store it as 010011010110000101101110. When a Consumer gets this message, it will deserialize back to Man.

    However, according to Confluent document :

    Data formats - The REST Proxy can read and write data using JSON, raw bytes encoded with base64 or using JSON-encoded Avro.

    enter image description here

    Therefore, a Producer using REST Proxy will change the message Man into TWFu (base64 encode) and send this to a Broker, and a Consumer using REST Proxy will base64 decode this back to Man.

    enter image description here