ZMQ Clients - Server - Worker Performance

I'am currently developing a system with a server, which gets tasks of some clients and processes them.

As I need high throughput I examined the speed for roundtrips/s in a 1Gbit network.

The scenario:

Clients have a Dealer socket and send a task to the server, server receives the tasks with a Router socket (TCP-Connection)
The server fowards the Tasks to one Worker-Thread (Dealer - Dealer via INPROC)
The worker thread processes the tasks and sends the response back on the same way

I used 16 Clients on 16 machines for the benchmark.

Benchmark 1: (without workers, server sends message directly back to the clients) Result:

1B-Messages: 143540,67 RT/s (RT = roundtrip)
10B-Messages: 140160,72 RT/s
100B-Messages: 129634,43 RT/s
500B-Messages: 120977,5 RT/s
1024B-Messages: 107983,59 RT/s

Benchmark 2: (with 10 work0ers, server just acts as a broker)

1B-Messages: 92873,51 RT/s (RT = roundtrip)
10B-Messages: 81619,33 RT/s
100B-Messages: 83606,02 RT/s
500B-Messages: 75229,45 RT/s
1024B-Messages: 63648,32 RT/s

Sources:

http://pastebin.com/5JBD0DUH (Benchmark 1)
http://pastebin.com/3iq2Bxqc (Benchmark 2)

Can someone help me identify why the TP is going down that much with just adding an inproc-roundtrip with some worker thread? I really expected a higher TP with the worker threads. Is the ZMQ Inproc Performance not that fast?

Solution

You don't use correct pattern for ZMQ_ROUTER/ZMQ_DEALER socket.

Pseudo code is

front = zmq_socket (context, ZMQ_ROUTER);
zmq_bind(front, "tcp://*:15555");
back = zmq_socket(context, ZMQ_DEALER);
zmq_bind (back, "inproc://abc");
// Here create thread
zmq_proxy(front, back, NULL); // zmq_proxy will not return

Pseudo code for thread is

socket = zmq_socket(context, ZMQ_REP);
zmq_connect(socket, "inproc://abc");
do {
   zmq_recv(...)
   zmq_send(...)
} while (1);

Pseudo code for client is

socket = zmq_socket(context, ZMQ_REQ);
zmq_connect(socket, "tcp://127.0.0.1:15555");
do {
   zmq_send(...)
   zmq_recv(...)
} while (1);