I am trying to enable session replication in my Wildfly 10.1 application with distributable WARs.
I am running on 2 instances of RedHat 7.2 on a managed host provider with full access to the OS and firewall. I don't have access to the router which our traffic is served, however the host has confirmed that multicast UDP is enabled.
I have SeLinux set to minimum, the ports are open in iptables, the multicast IPs have been subscribed, and my wildfly domain mode configuration is using a cloned full-ha profile with full-ha-sockets:
Here is the domain profile, which is vanilla with the exception of datasources:
<profile name="ha-dev2">
<subsystem xmlns="urn:jboss:domain:logging:3.0">
<add-logging-api-dependencies value="false"/>
<console-handler name="CONSOLE">
<level name="INFO"/>
<formatter>
<named-formatter name="COLOR-PATTERN"/>
</formatter>
</console-handler>
<periodic-rotating-file-handler name="FILE" autoflush="true">
<formatter>
<named-formatter name="PATTERN"/>
</formatter>
<file relative-to="jboss.server.log.dir" path="server.log"/>
<suffix value=".yyyy-MM-dd"/>
<append value="true"/>
</periodic-rotating-file-handler>
<logger category="com.arjuna">
<level name="WARN"/>
</logger>
<logger category="org.jboss.as.config">
<level name="DEBUG"/>
</logger>
<logger category="sun.rmi">
<level name="WARN"/>
</logger>
<root-logger>
<level name="INFO"/>
<handlers>
<handler name="CONSOLE"/>
<handler name="FILE"/>
</handlers>
</root-logger>
<formatter name="PATTERN">
<pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
</formatter>
<formatter name="COLOR-PATTERN">
<pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
</formatter>
</subsystem>
<subsystem xmlns="urn:jboss:domain:batch-jberet:1.0">
<default-job-repository name="in-memory"/>
<default-thread-pool name="batch"/>
<job-repository name="in-memory">
<in-memory/>
</job-repository>
<thread-pool name="batch">
<max-threads count="10"/>
<keepalive-time time="30" unit="seconds"/>
</thread-pool>
</subsystem>
<subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>
<subsystem xmlns="urn:jboss:domain:datasources:4.0">
<datasources>
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>
<!-- DATASOURCES REDACTED -->
<drivers>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
</subsystem>
<subsystem xmlns="urn:jboss:domain:ee:4.0">
<spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
<concurrent>
<context-services>
<context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
</context-services>
<managed-thread-factories>
<managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
</managed-thread-factories>
<managed-executor-services>
<managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" keepalive-time="5000"/>
</managed-executor-services>
<managed-scheduled-executor-services>
<managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" keepalive-time="3000"/>
</managed-scheduled-executor-services>
</concurrent>
<default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" jms-connection-factory="java:jboss/DefaultJMSConnectionFactory" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:ejb3:4.0">
<session-bean>
<stateless>
<bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
</stateless>
<stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>
<singleton default-access-timeout="5000"/>
</session-bean>
<mdb>
<resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:activemq-ra.rar}"/>
<bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
</mdb>
<pools>
<bean-instance-pools>
<strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
<strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
</bean-instance-pools>
</pools>
<caches>
<cache name="simple"/>
<cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>
</caches>
<passivation-stores>
<passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>
</passivation-stores>
<async thread-pool-name="default"/>
<timer-service thread-pool-name="default" default-data-store="default-file-store">
<data-stores>
<file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>
</data-stores>
</timer-service>
<remote connector-ref="http-remoting-connector" thread-pool-name="default"/>
<thread-pools>
<thread-pool name="default">
<max-threads count="10"/>
<keepalive-time time="100" unit="milliseconds"/>
</thread-pool>
</thread-pools>
<iiop enable-by-default="false" use-qualified-name="false"/>
<default-security-domain value="other"/>
<default-missing-method-permissions-deny-access value="true"/>
<log-system-exceptions value="true"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:io:1.1">
<worker name="default"/>
<buffer-pool name="default"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
<transport lock-timeout="60000"/>
<replicated-cache name="default" mode="SYNC">
<transaction mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
<distributed-cache name="concurrent" mode="SYNC" l1-lifespan="0" owners="2">
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<invalidation-cache name="entity" mode="SYNC">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps" mode="ASYNC"/>
</cache-container>
</subsystem>
<subsystem xmlns="urn:jboss:domain:iiop-openjdk:1.0">
<orb socket-binding="iiop" ssl-socket-binding="iiop-ssl"/>
<initializers security="identity" transactions="spec"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
<subsystem xmlns="urn:jboss:domain:jca:4.0">
<archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
<bean-validation enabled="true"/>
<default-workmanager>
<short-running-threads>
<core-threads count="50"/>
<queue-length count="50"/>
<max-threads count="50"/>
<keepalive-time time="10" unit="seconds"/>
</short-running-threads>
<long-running-threads>
<core-threads count="50"/>
<queue-length count="50"/>
<max-threads count="50"/>
<keepalive-time time="10" unit="seconds"/>
</long-running-threads>
</default-workmanager>
<cached-connection-manager/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
<subsystem xmlns="urn:jboss:domain:jgroups:4.0">
<channels default="ee">
<channel name="ee" stack="udp"/>
</channels>
<stacks>
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="MPING" socket-binding="jgroups-mping"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
</stacks>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jmx:1.3">
<expose-resolved-model/>
<expose-expression-model/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jpa:1.1">
<jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jsf:1.0"/>
<subsystem xmlns="urn:jboss:domain:jsr77:1.0"/>
<subsystem xmlns="urn:jboss:domain:mail:2.0">
<mail-session name="default" jndi-name="java:jboss/mail/Default">
<smtp-server outbound-socket-binding-ref="mail-smtp"/>
</mail-session>
</subsystem>
<subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
<server name="default">
<cluster password="${jboss.messaging.cluster.password:@password@}"/>
<bindings-directory/>
<journal-directory/>
<large-messages-directory/>
<paging-directory/>
<security-setting name="#">
<role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
</security-setting>
<address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
<http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
<http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
<param name="batch-delay" value="50"/>
</http-connector>
<in-vm-connector name="in-vm" server-id="0"/>
<http-acceptor name="http-acceptor" http-listener="default"/>
<http-acceptor name="http-acceptor-throughput" http-listener="default">
<param name="batch-delay" value="50"/>
<param name="direct-deliver" value="false"/>
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
<broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="http-connector"/>
<discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>
<cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
<jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
<jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
<connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
<connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
<pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
</server>
</subsystem>
<subsystem xmlns="urn:jboss:domain:modcluster:2.0">
<mod-cluster-config advertise-socket="modcluster" balancer="dev-ha-server-group" connector="ajp">
<dynamic-load-provider>
<load-metric type="busyness"/>
</dynamic-load-provider>
</mod-cluster-config>
</subsystem>
<subsystem xmlns="urn:jboss:domain:naming:2.0">
<remote-naming/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
<subsystem xmlns="urn:jboss:domain:remoting:3.0">
<http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:resource-adapters:4.0"/>
<subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>
<subsystem xmlns="urn:jboss:domain:sar:1.0"/>
<subsystem xmlns="urn:jboss:domain:security:1.2">
<security-domains>
<security-domain name="other" cache-type="default">
<authentication>
<login-module code="Remoting" flag="optional">
<module-option name="password-stacking" value="useFirstPass"/>
</login-module>
<login-module code="RealmDirect" flag="required">
<module-option name="password-stacking" value="useFirstPass"/>
</login-module>
</authentication>
</security-domain>
<security-domain name="jboss-web-policy" cache-type="default">
<authorization>
<policy-module code="Delegating" flag="required"/>
</authorization>
</security-domain>
<security-domain name="jboss-ejb-policy" cache-type="default">
<authorization>
<policy-module code="Delegating" flag="required"/>
</authorization>
</security-domain>
<security-domain name="jaspitest" cache-type="default">
<authentication-jaspi>
<login-module-stack name="dummy">
<login-module code="Dummy" flag="optional"/>
</login-module-stack>
<auth-module code="Dummy"/>
</authentication-jaspi>
</security-domain>
</security-domains>
</subsystem>
<subsystem xmlns="urn:jboss:domain:security-manager:1.0">
<deployment-permissions>
<maximum-set>
<permission class="java.security.AllPermission"/>
</maximum-set>
</deployment-permissions>
</subsystem>
<subsystem xmlns="urn:jboss:domain:singleton:1.0">
<singleton-policies default="default">
<singleton-policy name="default" cache-container="server">
<simple-election-policy/>
</singleton-policy>
</singleton-policies>
</subsystem>
<subsystem xmlns="urn:jboss:domain:transactions:3.0">
<core-environment>
<process-id>
<uuid/>
</process-id>
</core-environment>
<recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:undertow:3.1">
<buffer-cache name="default"/>
<server name="default-server">
<ajp-listener name="ajp" socket-binding="ajp" max-post-size="26214400"/>
<http-listener name="default" socket-binding="http" max-post-size="26214400" redirect-socket="https" enable-http2="true"/>
<https-listener name="https" socket-binding="https" max-post-size="26214400" security-realm="ApplicationRealm" enable-http2="true"/>
<host name="default-host" alias="localhost">
<location name="/" handler="welcome-content"/>
<filter-ref name="server-header"/>
<filter-ref name="x-powered-by-header"/>
<filter-ref name="gzipFilter" predicate="exists['%{o,Content-Type}'] and regex[pattern='(?:application/javascript|text/css|text/html|text/xml|application/json)(;.*)?', value=%{o,Content-Type}, full-match=true]"/>
</host>
</server>
<servlet-container name="default">
<jsp-config trim-spaces="true"/>
<websockets/>
</servlet-container>
<handlers>
<file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
</handlers>
<filters>
<response-header name="server-header" header-name="Server" header-value="WildFly/10"/>
<response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
<gzip name="gzipFilter"/>
</filters>
</subsystem>
<subsystem xmlns="urn:jboss:domain:webservices:2.0">
<wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
<endpoint-config name="Standard-Endpoint-Config"/>
<endpoint-config name="Recording-Endpoint-Config">
<pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
<handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
</pre-handler-chain>
</endpoint-config>
<client-config name="Standard-Client-Config"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:weld:3.0"/>
</profile>
This the socket binding group:
<socket-binding-group name="dev-full-ha-sockets" default-interface="public">
<socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
<socket-binding name="http" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:18443}"/>
<socket-binding name="iiop" interface="unsecure" port="3528"/>
<socket-binding name="iiop-ssl" interface="unsecure" port="3529"/>
<socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
<socket-binding name="jgroups-tcp" port="7600"/>
<socket-binding name="jgroups-tcp-fd" port="57600"/>
<socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
<socket-binding name="jgroups-udp-fd" port="54200"/>
<socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
</socket-binding-group>
And finally, the server-group:
<server-group name="dev-ha-server-group" profile="ha-dev2">
<jvm name="default">
<heap size="256m" max-size="1024m"/>
</jvm>
<socket-binding-group ref="dev-full-ha-sockets"/>
<deployments>
<!-- Deployments redacted -->
</deployments>
</server-group>
Using TCPDump, I can see traffic for modcluster from both of my servers.
I have also run the McastReceiverTest (src: https://github.com/belaban/JGroups/blob/master/tests/other/org/jgroups/tests/McastReceiverTest.java) via:
java org.jgroups.tests.McastReceiverTest -mcast_addr 230.0.0.4 -port 45700
and have used the following command:
printf "GET / HTTP/1.0\r\n\r\n" | nc -vu 230.0.0.4 45700
and see the UDP traffic coming through.
I am not seeing anything for the jgroups from wildfly though. In my logs, when I look for the subscribed channels, I see
"Received new cluster view for channel server: [app-one:server-one|0] (1) [app-one:server-one]"
But I never see my second server on either instance. When I test, I have an F5 load balancer in front of both RedHat instances.
I have a few assumptions that I am hoping to clear up:
Do I need to have Apache enabled for Modcluster? I'd assume not as it appears to be used for load balancing if you don't have a separate balancing mechanism.
Is Session Replication the same as running a "hot-hot" scenario so that 2 applications can run and keep a session alive? My understanding is that I should be able to log into the application on server A and server B would be able to keep that session alive as well. We are trying to prevent the need for session stickyness.
Can this run in a multi-server environment or is it only for 2 instances on the same local machine? I would assume that it should not matter if this is across a local machine or on 2 separate physicals/vms behind the same firewall.
Thank you.
I found a solution which has been eluding me. The problem with my configuration was in my bind IP. I was using 127.0.0.1 for my jboss.bind.address as I had Apache in front of my Wildfly configuration. I changed the bind IP to the IP of the NIC (in my case it was a 10.* address).
The problem with my configuration was in my bind IP. I was using 127.0.0.1 for my jboss.bind.address as I had Apache in front of my Wildfly configuration. I changed the bind IP to the IP of the NIC (in my case it was a 10.* address).