We have a Spring MVC web app (WAR) deploying to Tomcat (6.0.35) that launches a thread inside a separate JVM at deploy time (don't ask why - not my design) and then communicates with that thread via RMI over port 8888.
Despite being totally convoluded, this was working perfectly fine up until yesterday, and now the thread is failing at startup and despite our best efforts to add logging into the mix, we are hitting a wall. This is the only exception we are able to find in the logs:
Jun 12, 2012 3:11:36 AM com.ourapp.ImageController destroy
SEVERE: Shutdown Error: Lookup of RMI stub failed; nested exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused
Jun 12, 2012 3:11:37 AM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class org.springframework.web.context.ContextLoaderListener
java.lang.NoClassDefFoundError: org/springframework/web/context/ContextCleanupListener
at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:80)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3973)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4577)
at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1165)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1271)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.springframework.web.context.ContextCleanupListener
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1387)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
... 12 more
The ImageController is the Spring MVC Controller that is responsible for kicking off this daemon/spawned RMI thread. Based on the verbage of this error, does anybody have any idea what might be causing this "connection refused" error?
Running a netstat -an | grep 8888 (this is a Linux machine) produces no output which means nothing is listening on that port. Thanks in advance for any ideas/suggestions that lead to a fix.
Edit: Here's another ConnectionException we're seeing:
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 74 more
I'll move my comments to this answer.
Everything in your logs suggests that the problem is on your other machine where you connect to on port 8888. The netstat results prove that there's no process listening on that port - hence connection errors in your logs.
Next step I would take is check whether the application on the other machine (or in other JVM) is running at all (e.g. ps ax|grep java). And if it does - check its logs for clues why it failed to start the RMI service, or if it doesn't - start it.
Typically if an application can not open a socket, it's due to the fact that some other application is already using the port. It's not the case in your situation. It could be a case that two instances of the same application have been started: first application took 8888 port, second application couldn't take 8888 and failed, and then first application was shut down. After all you end up with nobody listening on the port.