I have a BPS process that rapidly performs a lot of calls to an ESB proxy. The ESB proxy calls an ESB API, which calls the API of our own product.
Often, the BPS process pauses for exactly 3 minutes while waiting for a response from the ESB. After the 3 minutes, everything continues as if nothing happened. Using Wireshark and some additional Log mediators, I have discovered the following:
The fact that this hanging takes exactly 3 minutes should be a hint. I searched the config files for anything specifying 180 seconds, but the only ones are the http socket timeouts and the http transportReceiver's threadKeepAliveTime. If I reduce those, the BPS throws a p2p communication error instead of continuing. To me, this indicates that something somewhere in the background is still causing a 3 minute delay for some calls.
By the way, the hanging does not occur for some specific calls only. I have been testing several times, performing the exact same calls each time, and I cannot predict which call will hang or even if any call will hang at all.
I ended up getting WSO2 support involved in this. Apparently, I was hitting some very rare edge case which was very hard for the WSO2 engineers to reproduce.
In the end, the fix turned out to be adding the following property just before the send mediator in the proxy service:
<property name="NO_KEEPALIVE" value="true" scope="axis2"/>