[SOLVED] Netty: how do I reduce delay between consecutive messages from the server?

Netty: how do I reduce delay between consecutive messages from the server?

I'm on the dev team for a socket server which uses Netty. When a client sends a request, and the server sends a single response, the round trip time is quite fast. (GOOD) We recently noticed that if the request from the client triggers two messages from the server, even though the server writes both messages to the client at about the same time, there is a delay of more than 200ms between the first and second message arriving on the remote client. When using a local client the two messages arrive at the same time. If the remote client sends another request before the second message from the server arrives, that second message is sent immediately, but then the two messages from the new request are both sent with the delay of over 200ms.

Since it was noticed while using Netty 3.3.1, I tried upgrading to Netty 3.6.5 but I still see the same behavior. We are using NIO, not OIO, because we need to be able to support large numbers of concurrent clients.

Is there a setting that we need to configure that will reduce that 200+ ms delay?

editing to add a code snippet. I hope these are the most relevant parts.

@Override
public boolean openListener(final Protocol protocol,
        InetSocketAddress inetSocketAddress) throws Exception {
    ChannelFactory factory = new NioServerSocketChannelFactory(
            Executors.newCachedThreadPool(),
            Executors.newCachedThreadPool(),
            threadingConfiguration.getProcessorThreadCount());

    ServerBootstrap bootstrap = new ServerBootstrap(factory);

    final ChannelGroup channelGroup = new DefaultChannelGroup();
    bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
               .... lots of pipeline setup snipped ......
    });

    Channel channel = bootstrap.bind(inetSocketAddress);

    channelGroup.add(channel);

    channelGroups.add(channelGroup);

    bootstraps.add(bootstrap);

    return true;
}

The writer factory uses ChannelBuffers.dynamicBuffer(defaultMessageSize) for the buffer, and when we write a message it's Channels.write(channel, msg).

What else would be useful? The developer who migrated the code to Netty is not currently available, and I'm trying to fill in.

Solution

200ms strikes me as the magic number of the Nagle's algorithm. Try setting the TcpNoDelay to true on both sides.

This is how you set the option for the server side.

    serverBootstrap.setOption("child.tcpNoDelay", true);

This is for the client side.

    clientBootStrap.setOption("tcpNoDelay", true);

Further reading: http://www.stuartcheshire.org/papers/NagleDelayedAck/