zeromqpublish-subscribedistributed-computingpyzmqlow-latency

What is the ZeroMQ PUB/SUB internal behaviour?


I'm trying to get my head around to the behaviour of zmq with PUB/SUB.

Q1: I can't find a real reason why with the PUSH/PULL sockets combo I can create a queue that actually queue in memory messages that it can't get delivered (the consumer is not available) when with the PUB/SUB not.

Q2: Is there any technical whitepaper or document that describes in detail the internals of the sockets?

EDIT: This example of PUSH/PULL streamer works as expected (the worker join late or restart and gets the queued messages in the feeder. PUB/SUB forwarder does not behave in the same way.


Solution

  • While Q1 is hard to be answered / fully addressed without a SLOC ...

    there is still a chance that in your code you just forgot to set a subscription topic-filter:

    aSubSOCKET.setsockopt( zmq.SUBSCRIBE = "" )          # ->recv "EVERYTHING" / NO-TOPIC-FILTER
    aSubSOCKET.setsockopt( zmq.SUBSCRIBE = "GOOD-NEWS" ) # ->recv "GOOD-NEWS" MESSAGES to be received only
    

    Yes, there are exhaustive descriptions of all ZeroMQ API calls +

    besides the API manpage collection for ØMQ/2.1.1 and other versions, there is a great online published PDF book "Code Connected, Vol.1" from Pieter Hintjens himself.

    It is worth reading. There's a lot of insights into general distributed-processing area and the ZeroMQ way.