Is it mandatory that all the messages in a Kafka batch should go to the same partition?
Or is it possible to batch up messages (which are supposed to go to different partitions) together?
It depends on the producer's partition assignment algorithm. The default is RoundRobin so it will be split equally between the partitions.
You can pass a key for every record in the batch and then a default hash function will be executed for each key and the broker will put the record in the partition returned from the hash function.
You can also override the hash function and write one yourself.
Otherwise, if your Kafka client supports you can pass for each record the partition explicitly.
But you don't have to group messages by partition - only by topic.