When I work in WSO2 I have seen some customers are facing the below error in their APIM Gateway nodes intermittently.
"ERROR {org.wso2.andes.transport.network.mina.MinaNetworkHandler} - Exception caught by Mina java.io.IOException: Connection timed out"
This error could occur when there is a firewall between the Gateway node and Traffic Manager(TM) node and connections getting into an idle state because of the network inactivity. In such Idle states, the firewall breaks the connection, hence this error can occur.
In order to prevent this issue, we need to keep the connection in an active state. Therefore, we can achieve this by establishing a heartbeat between two servers. Once the heartbeat is configured in the client, it periodically sends small frames to the server in order to verify the connection status. Since this is a small frame, it won't generate considerable network traffic. Please follow the below steps to configure the heartbeat value as 60 seconds.
Update the <APIM_HOME>/repository/conf/deployment.toml file as below of the Traffic Manager nodes.
[qpid.heartbeat]
delay = 60
timeout_factor = 2.0
Restart all the APIM-TM nodes and gateway nodes.
Ref: https://apim.docs.wso2.com/en/latest/reference/config-catalog/#qpid-broker-heartbeat-configurations