spring hessian client socket connection reset

I am using spring 3.1.0, com.springsource.com.caucho-3.2.1.jar and tomcat-6.0.33 both sides (client/server). All the remote service calls are working fine without any issues except long time taking services(more then 9/10 minutes).

I am using Spring-Security to protect remote calls.

I have created a new remote service that takes approx 30 minutes to make a response to client in real schenerio. The service works perfectly if time taken of execution is less then 9.xx/10 minutes but after reaching to 9.xx/10 minutes, I am getting Connection reset on my Hessian Client.

Client configuration

<bean id="someService" class="org.springframework.remoting.caucho.HessianProxyFactoryBean">
    <property name="serviceUrl" value="http://hostname:8080/remoting/someService"/>
    <property name="serviceInterface" value="com.SomeService"/>
    <property name="username" value="${service.username}"/>
    <property name="password" value="${service.password}"/>
</bean>

Server configuration

<bean id="someService" class="com.SomeService" />

<bean name="/someService" class="org.springframework.remoting.caucho.HessianServiceExporter">
    <property name="service" ref="someService" />
    <property name="serviceInterface" value="com.SomeService" />
</bean>

Client side - Stack Trace:

2013-Feb-28 17:48:19 DEBUG [http-8080-1] com.SomeService:85 - Calling Some Service
2013-Feb-28 17:58:16 ERROR [http-8080-1] com.SomeService:113 - 
org.springframework.remoting.RemoteConnectFailureException: 
Cannot connect to Hessian remote service at [http://hostname:8080/remoting/someService]; 
nested exception is com.caucho.hessian.client.HessianConnectionException: 
500: java.net.SocketException: Connection reset
at org.springframework.remoting.caucho.HessianClientInterceptor.convertHessianAccessException(HessianClientInterceptor.java:262)

Server side - Tomcat (localhost.log)

SEVERE: Servlet.service() for servlet [remoting] in context with path [/JTService] 
threw exception [Hessian skeleton invocation failed; nested exception is
ClientAbortException:  java.net.SocketException: Connection reset] with root cause
java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)

I tried to set 30 minutes value of readTimeout available under HessianProxyFactoryBean but getting the same exception after 9.xx/10 minutes. However when i tried with readTimeout 2 minutes I am getting Read timed out after 2 minutes.

After setting readTimeout to 2 minutes I am getting :

2013-Feb-28 17:44:10 ERROR [http-8080-1] com.SomeService:113 - org.springframework.remoting.RemoteConnectFailureException: 
Cannot connect to Hessian remote service at [http://hostname:8080/remoting/someService]; 
nested exception is com.caucho.hessian.client.HessianConnectionException: 500: java.net.SocketTimeoutException: Read timed out
org.springframework.remoting.RemoteConnectFailureException: 
Cannot connect to Hessian remote service at [http://hostname:8080/remoting/someService]; 
nested exception is com.caucho.hessian.client.HessianConnectionException: 500: java.net.SocketTimeoutException: Read timed out

As Like readTimeout I did'nt see and setting related to connection timeout under HessianProxyFactoryBean.

Please suggest what to do?

After some debugging, I found that my Linux / CentOS server was closing the network connection after some minutes. Connection timeout issue resolved after increasing connection keepalive time under */proc/sys/net/ipv4/tcp_keepalive_time*.

Linux console sample output :

Shows keepalive time in seconds

[root@hostname ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time 
7200

Use following command to increase the value

[root@hostname ~]# echo "21600" > /proc/sys/net/ipv4/tcp_keepalive_time

Hope this helps others too