[SOLVED] How we can write single message(not batch) fast in kafka?

How we can write single message(not batch) fast in kafka?

I am new to Golang and Kafka and I am using segmentio kafka-go to connect to Kafka server using Golang. As of now I want to push every event of user in Kafka, so I want to push single message(and not in batch), but since the write operation provided by this library takes same time for either batch or single message, it is taking a lot of time. Is there any way of writing single message fast so that i can push million events in kafka in less time?

I have tested it for single message, and batch messages, it is taking same time (min was 10ms).

Solution

I think your problem is just the WriterConfig.

For example, if your config looks like the example on segmentio/kafka-go docs:

w := kafka.NewWriter(kafka.WriterConfig{
    Brokers:      []string{"localhost:9092"},
    Topic:        "topic-A",
    Balancer:     &kafka.LeastBytes{},
})

You could try setting batch size and batch timeout:

w := kafka.NewWriter(kafka.WriterConfig{
    Brokers:      []string{"localhost:9092"},
    Topic:        "topic-A",
    Balancer:     &kafka.LeastBytes{},
    BatchSize:    1,
    BatchTimeout: 10 * time.Millisecond,
})

It happens because kafka-go waits by default 1 second until the batch reach the maximum size, which is by default 100 messages, as we can see in the code.

Hope it helps you.

Update: Be aware that sending the messages one by one slows the process. For example: sending 100 messages in batch took on my computer 0.0107s. Sending the same 100 messages one by one took 0.0244s.