JMS master/backup failover and failback configuration in Red Hat 7.2 EAP

We use Red Hat 7.2 EAP. We have X number of producers on Y number of machines sending messages to a JMS queue on a remote machine. The JMS queue is configured with a master and backup. X number of consumers on Y number of machines are consuming messages from the queue. The master and backup are on different machines. When the master goes down the producers and consumers need to continue processing messages. When the master comes back the consumers and producers need to fail back to the master.

I have read the documents published by Red Hat multiple times and best I can tell this should be easily done. However, when the consumers or producers start up and the master is down they do not have any way to connect to the backup queue. How do people handle this situation? Do the consumers and the producers have to be programed to know about the master and the backup and then try to connect to the backup if the master is not available or fails? The documentation leads me to believe that failover should be automatic in that the master communicates the backup location to client when it first attaches. If there are dozens of machines involved how does the configuration of the master/backup get to the consumers and producers? Do each of the consumer/producer nodes have to be configured with IP addresses? How do people make this type of thing scale?

We currently set up the producers and consumers to connect like this:

final Properties env = new Properties();
env.put(Context.INITIAL_CONTEXT_FACTORY, INITIAL_CONTEXT_FACTORY);
env.put(Context.PROVIDER_URL, System.getProperty(Context.PROVIDER_URL, PROVIDER_URL));
env.put(Context.SECURITY_PRINCIPAL, DEFAULT_USERNAME);
env.put(Context.SECURITY_CREDENTIALS, DEFAULT_PASSWORD);
namingContext = new InitialContext(env);

EAP uses a client/server JNDI implementation and JNDI connections are 100% independent of JMS connections. Since JNDI is the starting point for your connectivity with EAP you'll want to tackle that first. Typically an EAP JNDI URL looks something like this:

http-remoting://host:8080

Of course, if host is unavailable then the JNDI lookup will fail. You can address this in a number of ways the most basic of which is specifying multiple URLs, e.g.:

http-remoting://host1:8080,http-remoting://host2:8080

Other solutions involve dedicated load-balancers, DNS-based redirection, etc.

Once the JNDI lookup is complete you'll be able to establish a JMS connection. Assuming the server pair is set up correctly the connection will then fail-over and fail-back as needed.

Here is the messaging subsystem configuration from the default standalone-full.xml that ships with EAP:

        <subsystem xmlns="urn:jboss:domain:messaging-activemq:13.0">
            <server name="default">
                <statistics enabled="${wildfly.messaging-activemq.statistics-enabled:${wildfly.statistics-enabled:false}}"/>
                <security-setting name="#">
                    <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                </security-setting>
                <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10"/>
                <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
                <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
                    <param name="batch-delay" value="50"/>
                </http-connector>
                <in-vm-connector name="in-vm" server-id="0">
                    <param name="buffer-pooling" value="false"/>
                </in-vm-connector>
                <http-acceptor name="http-acceptor" http-listener="default"/>
                <http-acceptor name="http-acceptor-throughput" http-listener="default">
                    <param name="batch-delay" value="50"/>
                    <param name="direct-deliver" value="false"/>
                </http-acceptor>
                <in-vm-acceptor name="in-vm" server-id="0">
                    <param name="buffer-pooling" value="false"/>
                </in-vm-acceptor>
                <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
                <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
                <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
                <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
            </server>
        </subsystem>

Here you can see multiple jms-queue and connection-factory elements with their respective JNDI bindings in the entries attribute.

If you have messaging clients running in remote instances of EAP and you want their JNDI lookups to be local then you can configure an external-context in the naming subsystem. This will effectively turn local lookups into remote ones. For example:

<subsystem xmlns="urn:jboss:domain:naming:2.0">
  <bindings>
    <external-context name="java:global/jms" module="org.jboss.as.naming" class="javax.naming.directory.InitialDirContext" cache="true">
      <environment>
        <property name="java.naming.factory.initial" value="org.wildfly.naming.client.WildFlyInitialContextFactory"/>
        <property name="java.naming.provider.url" value="http-remoting://host1:8080,http-remoting://host2:8080"/>
        <property name="java.naming.security.principal" value="user"/>
        <property name="java.naming.security.credentials" value="pass"/>
      </environment>
    </external-context>
  </bindings>
</subsystem>

See the documentation for more details.