ActiveMQ failover timeout works randomly

I'm facing an issue with the timeout failover option. Basically I want a client that keeps sending message to a broker, if the broker is down, it will try to reconnect until broker is up again. At the same time, the send will timeout, so it won't pending there forever.

However, after some tests, it seems the timeout option does not work properly. Sometimes, it does timeout, but sometimes, it just hangs there. I know another option is maxReconnectAttempt. But the thing is I want it to try to reconnect forever.

Below is the url I am using:

failover:(tcp://10.5.0.198:61616)?timeout=1000

I have a two-server system. One works as the broker, and one works as the client. When I toggle the broker to start/stop, I get the log message below:

We sent a Message!
2016-02-24 20:29:06,967 [198:61616@17075] - WARN  FailoverTransport - Transport (tcp://10.5.0.198:61616) failed, reason:  java.io.EOFException, attempting to automatically reconnect
Failover timeout of 1000 ms reached.
Failover timeout of 1000 ms reached.
Failover timeout of 1000 ms reached.
Failover timeout of 1000 ms reached.
Failover timeout of 1000 ms reached.
Failover timeout of 1000 ms reached.
We sent a Message!
We sent a Message!
We sent a Message!
We sent a Message!
We sent a Message!
2016-02-24 20:29:48,688 [198:61616@17099] - WARN  FailoverTransport - Transport (tcp://10.5.0.198:61616) failed, reason:  java.io.EOFException, attempting to automatically reconnect
We sent a Message!
We sent a Message!
We sent a Message!
We sent a Message!
We sent a Message!
2016-02-24 21:57:50,777 [198:61616@18542] - WARN  FailoverTransport - Transport (tcp://10.5.0.198:61616) failed, reason:  java.io.EOFException, attempting to automatically reconnect
Failover timeout of 1000 ms reached.
Failover timeout of 1000 ms reached.
Failover timeout of 1000 ms reached.

As you can see, the second time when the broker was down, it didn't timeout for more than one hour.

Below is the code I am using:

private final String connectionUri = "failover:(tcp://10.5.0.198:61616)?timeout=1000";
private ActiveMQConnectionFactory connectionFactory;
private Connection connection;
private Session session;
private Destination destination;

public void before() throws Exception {
    connectionFactory = new ActiveMQConnectionFactory(connectionUri);
    connection = connectionFactory.createConnection();
    connection.start();
    session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
    destination = session.createQueue("MyQueue");
}

public void after() throws Exception {
    if (connection != null) {
        connection.close();
    }
}

public void run() throws Exception {

    MessageProducer producer = session.createProducer(destination);
    try {
        TextMessage message = session.createTextMessage();
        message.setText("We sent a Message!");
        producer.send(message);
    } finally {
        producer.close();
    }

    MessageConsumer consumer = session.createConsumer(destination);
    try {
        TextMessage message = (TextMessage) consumer.receive();
        System.out.println(message.getText());
    } finally {
        consumer.close();
    }
}
public static void main(String[] args) {
    SimpleJMS example = new SimpleJMS();
    System.out.print("\n\n\n");
    System.out.println("Starting SimpleJMS example now...");
    try {
        example.before();
        for(int i =0;i<1000;i++){
            Thread.sleep(1000);
            try{
                example.run();
            } catch (Exception e){
                System.out.println(e.getMessage());
            }
        }
        example.after();
    } catch (Exception e) {
        System.out.println("Caught an exception during the example: " + e.getMessage());
    }
    System.out.println("Finished running the SimpleJMS example.");
    System.out.print("\n\n\n");
}

I couldn't figure this out. After some research, it seems to be a problem with AMQ.

So in order to not block the whole thing, I switched to network broker which provide a local buffer so when it's doing the retry, the send operation will continue.

This is the best solution I can find for now.