dockerdocker-swarmactivemq-artemisjgroups

ActiveMQ Artemis cluster on Docker Swarm


I am trying to setup simple cluster of 2 ActiveMQ Artemis brokers within Docker Swarm environment.

Stack configuration

version: "3.7"

services:
    broker_1:
        image: broker-local
        build:
            context: ./activemq
            dockerfile: Dockerfile-centos
        ports:
            - "8161:8161" # Embedded web server
            - "61616:61616" # Main Artemis acceptor
        networks:
            artemis-cluster-network:
        environment:
            EXTRA_ARGS: "--http-host 0.0.0.0 --relax-jolokia"
            ANONYMOUS_LOGIN: "true"
        configs:
            -   source: broker_1_config
                target: /run/config/broker.xml

    broker_2:
        image: broker-local
        depends_on:
            - "broker_1"
        build:
            context: ./activemq
            dockerfile: Dockerfile-centos
        ports:
            - "8162:8161" # Embedded web server
            - "61617:61617" # Main Artemis acceptor
        networks:
            artemis-cluster-network:
        environment:
            EXTRA_ARGS: "--http-host 0.0.0.0 --relax-jolokia"
        configs:
            -   source: broker_2_config
                target: /run/config/broker.xml

networks:
    artemis-cluster-network:
        driver: overlay
        attachable: true

configs:
    broker_1_config:
        file: ./broker_1.xml
    broker_2_config:
        file: ./broker_2.xml

Broker_1 configuration

   <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
      xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
      <core xmlns="urn:activemq:core">

         <name>broker_1</name>

         <bindings-directory>./data/bindings</bindings-directory>

         <journal-directory>./data/journal</journal-directory>

         <large-messages-directory>./data/largemessages</large-messages-directory>

         <paging-directory>./data/paging</paging-directory>

         <!-- Connectors -->

         <connectors>
            <connector name="netty-connector">tcp://0.0.0.0:61616</connector>
            <!-- connector to the server1 -->
            <connector name="server2-connector">tcp://broker_2:61617</connector>
         </connectors>

         <!-- Acceptors -->
         <acceptors>
            <acceptor name="netty-acceptor">tcp://0.0.0.0:61616</acceptor>
         </acceptors>

         <cluster-connections>
            <cluster-connection name="my-cluster">
               <connector-ref>netty-connector</connector-ref>
               <retry-interval>500</retry-interval>
               <use-duplicate-detection>true</use-duplicate-detection>
               <message-load-balancing>STRICT</message-load-balancing>
               <max-hops>5</max-hops>
               <static-connectors allow-direct-connections-only="true">
                  <connector-ref>server2-connector</connector-ref>
               </static-connectors>
            </cluster-connection>
         </cluster-connections>

         <!-- Other config -->

         <security-settings>
            <!--security for example queue-->
            <security-setting match="exampleQueue">
               <permission roles="guest" type="createDurableQueue"/>
               <permission roles="guest" type="deleteDurableQueue"/>
               <permission roles="guest" type="createNonDurableQueue"/>
               <permission roles="guest" type="deleteNonDurableQueue"/>
               <permission roles="guest" type="consume"/>
               <permission roles="guest" type="send"/>
            </security-setting>
         </security-settings>

         <addresses>
            <address name="exampleQueue">
               <anycast>
                  <queue name="exampleQueue"/>
               </anycast>
            </address>
         </addresses>

         <broker-plugins>
            <broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
               <property key="LOG_ALL_EVENTS" value="true"/>
               <property key="LOG_CONNECTION_EVENTS" value="true"/>
               <property key="LOG_SESSION_EVENTS" value="true"/>
               <property key="LOG_CONSUMER_EVENTS" value="true"/>
               <property key="LOG_DELIVERING_EVENTS" value="true"/>
               <property key="LOG_SENDING_EVENTS" value="true"/>
               <property key="LOG_INTERNAL_EVENTS" value="true"/>
            </broker-plugin>
         </broker-plugins>
      </core>
   </configuration>

Broker_2 configuration

  <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
     xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
     <core xmlns="urn:activemq:core">

        <name>broker_2</name>

        <bindings-directory>./data/bindings</bindings-directory>

        <journal-directory>./data/journal</journal-directory>

        <large-messages-directory>./data/largemessages</large-messages-directory>

        <paging-directory>./data/paging</paging-directory>

        <!-- Connectors -->
        <connectors>
           <connector name="netty-connector">tcp://0.0.0.0:61617</connector>
           <!-- connector to the server0 -->
           <connector name="server1-connector">tcp://broker_1:61616</connector>
        </connectors>

        <!-- Acceptors -->
        <acceptors>
           <acceptor name="netty-acceptor">tcp://0.0.0.0:61617</acceptor>
        </acceptors>

        <cluster-connections>
           <cluster-connection name="my-cluster">
              <connector-ref>netty-connector</connector-ref>
              <retry-interval>500</retry-interval>
              <use-duplicate-detection>true</use-duplicate-detection>
              <message-load-balancing>STRICT</message-load-balancing>
              <max-hops>5</max-hops>
              <static-connectors allow-direct-connections-only="true">
                 <connector-ref>server1-connector</connector-ref>
              </static-connectors>
           </cluster-connection>
        </cluster-connections>

        <!-- Other config -->

        <security-settings>
           <!--security for example queue-->
           <security-setting match="exampleQueue">
              <permission roles="guest" type="createDurableQueue"/>
              <permission roles="guest" type="deleteDurableQueue"/>
              <permission roles="guest" type="createNonDurableQueue"/>
              <permission roles="guest" type="deleteNonDurableQueue"/>
              <permission roles="guest" type="consume"/>
              <permission roles="guest" type="send"/>
           </security-setting>
        </security-settings>

        <addresses>
           <address name="exampleQueue">
              <anycast>
                 <queue name="exampleQueue"/>
              </anycast>
           </address>
        </addresses>

        <broker-plugins>
           <broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
              <property key="LOG_ALL_EVENTS" value="true"/>
              <property key="LOG_CONNECTION_EVENTS" value="true"/>
              <property key="LOG_SESSION_EVENTS" value="true"/>
              <property key="LOG_CONSUMER_EVENTS" value="true"/>
              <property key="LOG_DELIVERING_EVENTS" value="true"/>
              <property key="LOG_SENDING_EVENTS" value="true"/>
              <property key="LOG_INTERNAL_EVENTS" value="true"/>
           </broker-plugin>
        </broker-plugins>
     </core>
  </configuration>

After running both services nothing happens. Nothing interesting is logged.

I tried UDP discovery approach but found out that is not possible on Docker Overlay Network.

Then I tried JGroups approach which did not work also. There were attempts to establish connection but every attempt timed out. And attempts were made to hosts represented by container ID. This approach could be interesting but shared ping directory which is used by JGRoups is not valid for Swarm approach. On local machine it was not problem to share it between two containers.

Also I am using Fiddler so names like broker_1 are translated to localhost.


Solution

  • The connector named "netty-connector" shouldn't use 0.0.0.0. This is the address that will be sent to other nodes in the cluster to tell them how to connect back to the node who sent it. The address 0.0.0.0 will be meaningless in that context. It needs to be the actual IP address or hostname where the broker is listening for network connections.