message-queuewan

Message queue over high latency WAN?


Is there a message queue system optimized for a WAN with >200ms RTT? E.g., sending messages from US to Asia.

I'm looking for a MQ that can batch messages together, pipeline lots of messages across multiple TCP connections, and/or use UDP to blast messages (like UDT).


Solution

  • There are two solutions which I can think of which fit the problem statement you have described, if you are using Kafka for message queueing within a single geographical location. Reliability of message delivery and latency optimizations would be 2 key goals for your project.

    Kafka community has a MirrorMaker tool (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330), which has been extended by Uber, in form of ureplicator (https://eng.uber.com/ureplicator/).