In production environment I got frequently thread hung error message.
Log is below.
[12/16/15 16:47:30:095 PHT] 00000030 ThreadMonitor W WSVR0605W: Thread "WebContainer : 11" (000005bb) has been active for 743576 milliseconds and may be hung. There is/are 5 thread(s) in total in the server that may be hung.
at java.util.HashMap.getEntry(HashMap.java:479)
at java.util.HashMap.get(HashMap.java:429)
at com.c2lbiz.symbiosys.insurance.product.business.ProductBO.getElementKeyString(ProductBO.java:222)
at com.c2lbiz.symbiosys.insurance.product.business.ProductBO.evaluateField(ProductBO.java:2156)
at com.c2lbiz.symbiosys.insurance.product.business.ProductBO.calculateFundFacets(ProductBO.java:2594)
at com.c2lbiz.symbiosys.insurance.product.business.ProductBO.calculateIllustration(ProductBO.java:1113)
at com.c2lbiz.symbiosys.insurance.quotation.util.QuotationUtil.generateIllustrationOutput(QuotationUtil.java:486)
at com.c2lbiz.symbiosys.insurance.quotation.business.QuoteIllustration.execute(QuoteIllustration.java:49)
at com.c2lbiz.symbiosys.insurance.controller.quotation.QuoteCaptureController.generateIllustrationPDF(QuoteCaptureController.java:176)
at com.c2lbiz.symbiosys.insurance.controller.quotation.GetOutputPreferencesController.process(GetOutputPreferencesController.java:66)
at com.c2lbiz.symbiosys.insurance.controller.quotation.QuoteCaptureController.process(QuoteCaptureController.java:1)
at com.c2lbiz.symbiosys.systemcommon.controller.BaseController.handleRequestInternal(BaseController.java:106)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:874)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:790)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:359)
at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)
at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)
at org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:271)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)
at org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)
at org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)
at org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)
at org.springframework.security.concurrent.ConcurrentSessionFilter.doFilterHttp(ConcurrentSessionFilter.java:99)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)
at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:174)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:967)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1107)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:919)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
It started from hanging thread 1 onward means
There is/are 1 thread(s) in total in the server that may be hung.
There is/are 2 thread(s) in total in the server that may be hung.
There is/are 3 thread(s) in total in the server that may be hung.
There is/are 4 thread(s) in total in the server that may be hung.
There is/are 5 thread(s) in total in the server that may be hung.
There is/are 6 thread(s) in total in the server that may be hung.
There is/are 7 thread(s) in total in the server that may be hung.
There is/are 8 thread(s) in total in the server that may be hung.
There is/are 9 thread(s) in total in the server that may be hung.
There is/are 10 thread(s) in total in the server that may be hung.
...............
...............
There is/are 77 thread(s) in total in the server that may be hung.
After that PDF generation stops working and batch running also stops running. Batch job generate pdf and send to provided emailid. What is root cause of this error? Same code is working fine on UAT and SIT environment.
Under some circumstances, a get() from an unsynchronized HashMap can cause an infinite loop. This can occur if a concurrent put() causes a rehash of the Map. In my case : I am using unsynchronized hashmap and concurrent thread calling get and put method on that hashmap. It caused infinite loop on get method and thread gets hung.It is my observation.