Usually, the order of messages between Consumers is not guaranteed, since this prohibits CAP-theorem.
Consistency in this theorem can be determined through the theory of Actors, that is, when we preserve the fact of receiving messages and their order - only for a specific Consumer.
An Actor itself is an entity that receives a Message, processes it atomically, and sends a message to the next Actor. Actors work asynchronously, do not transfer the application context among themselves and do not have direct access to an instance of another Actor. Each Actor knows nothing about the other factor.
In a distributed system, we cannot normally observe Consistency, and at the same time make the system fault-tolerant. Therefore, we cannot preserve the order of messages between Consumers.
However, Kafka introduced such a feature as the use of a transaction mechanism.
I read that Kafka now supports atomic writing in multiple partitions using new transactional APIs. This allows the producer to send message packets to multiple sections so that either all messages from the packet will be visible to any consumer, or none of them will be visible to anyone.
I don't understand some points :
If a transaction occurs within a single Topic, is the order of messages guaranteed (after all, if we rearrange operations within a transaction, we may get data inconsistency)?
If a transaction is distributed over several Topics (Aggregation of data from several unrelated tasks (Topics), is the ordering between transaction sub-operations guaranteed here as well?
How do such mechanisms work in general, does anyone just (can anyone explain with clear terms and examples ?) know the explanation of such phenomena?
In Kafka, messages are only ordered at the partition level. Transactions do not change this.
Transactions only ensure that messages inside are either all written or all discarded. If you write to multiple partitions (in one or multiple topics) in a transaction, only messages within the same partitions are guaranteed to be read in order.