pythonsocketszeromqnetmq

Conflate PUSH-PULL Socket between NetMQ and ZeroMQ


I wanted to create a socket between a C# and a Python application and ZeroMQ seemed like a good implementation for this. I want to send data bidirectional and non-blocking, so for the prototype I though I would make two PULL-PUSH sockets and this works fine.

Now, because the data is time-sensitive, I only want to keep the latest message and not spend time processing them in FIFO order. After looking around, I found the CONFLATE argument. Unfortunatelty this does not seem to work between ZeroMQ and NetMQ, as NetMQ does not have a formal definition of this argument.

After some testing (see image) it seems to work for the receiving side in Python, but not the otherway around. Does anybody have any information or workarounds for this? What would be the best and most efficient approach/ architecture?

Any help would be appreciated.

enter image description here


Solution

  • Q : "(...) any information or workarounds for this?"

    -- a cheapest trick would be to first try a hack to pass the actual value of the zmq.CONFLATE as a #define'd constant ( an integer from .h-file or from ZeroMQ RFC specification document ), for a sake if the ZeroMQ native tools, behind the NetMQ wrapped/published access to the actual API, still can turn to operate the actual native Socket-instance in the CONFLATE-mode internally, as documented in the ZeroMQ API. If so, we can celebrate and go get other stuff done.

    -- if not, try to still use the CONFLATE-mode at least on the non-NetMQ sides, as this may reduce at least the outgoing or incoming traffic from/to-there, if the CONFLATE-settings can detect and avoid any "old"-messages there, so efficiently not being stacked into the NetMQ ( so far not configurable by the wrapper as-is ) FIFO-queue towards the user-code.

    -- as a last resort on the ( so far in 2024-Q2 ) under-equipped NetMQ wrapper-side might serve a strategy to implement a fast, non-blocking & zero-copy .poll()-method driven read-ahead circular-buffer feeder, that will maintain the local PULL-socket ZeroMQ buffers off-loaded to this user-code circular-buffer and this will serve only "The Last" ( manually CONFLATE-ed ) message for the actual user-code processing. Zero-copy circular-buffer is just a pointer-passing fast off-loading trick, so maintaining just a such pointer-to-last-message reference updated, the rest is zero-copy, thanks to all the brave work of the ZeroMQ fathers and shall be fine.