I have two instances of Keycloak running on two different host machines (each keycloak instance is in a docker container). I followed the guide from this repo. I use JDBC_PING as the discovery protocol and I can see from the logs that two keycloak instances have discovered each other.
2023-07-26 19:39:44,839 INFO [org.infinispan.CLUSTER] (jgroups-7,vm200-9849) [Context=sessions] ISPN100010: Finished rebalance with members [vm200-9849, vm202-18591], topology id 5
2023-07-26 19:39:44,849 INFO [org.infinispan.CLUSTER] (jgroups-7,vm200-9849) [Context=work] ISPN100002: Starting rebalance with members [vm200-9849, vm202-18591], phase READ_OLD_WRITE_ALL, topology id 2
2023-07-26 19:39:44,850 INFO [org.infinispan.LIFECYCLE] (jgroups-7,vm200-9849) [Context=work] ISPN100002: Starting rebalance with members [vm200-9849, vm202-18591], phase READ_OLD_WRITE_ALL, topology id 2
2023-07-26 19:39:44,851 INFO [org.infinispan.LIFECYCLE] (non-blocking-thread--p2-t6) [Context=work] ISPN100010: Finished rebalance with members [vm200-9849, vm202-18591], topology id 2
2023-07-26 19:39:44,872 INFO [org.infinispan.CLUSTER] (jgroups-9,vm200-9849) [Context=work] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3
2023-07-26 19:39:44,875 INFO [org.infinispan.CLUSTER] (jgroups-9,vm200-9849) [Context=work] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4
2023-07-26 19:39:44,878 INFO [org.infinispan.CLUSTER] (jgroups-9,vm200-9849) [Context=work] ISPN100010: Finished rebalance with members [vm200-9849, vm202-18591], topology id 5
When I login to the two keycloak instances using different browsers (each browser has a different URL), I can see that are two admin sessions. Any users/clients I make on one instance is reflected on the other.
However, I cannot use the REST API Auth token generated from one instance to work on the other. I was under the assumption that in a clustered setup different keycloak instances should be able to work with the same auth token because I could not find any specific articles or documents relating to this.
Here is my infinispan.xml
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
xmlns="urn:infinispan:config:11.0">
<jgroups>
<stack name="postgres-jdbc-ping-tcp" extends="tcp">
<!-- <TCP external_addr="${env.JGROUPS_DISCOVERY_EXTERNAL_IP:127.0.0.1}" />
<TCP bind_port="7600" /> -->
<JDBC_PING
connection_driver="org.postgresql.Driver"
connection_username="${env.KC_DB_USERNAME}"
connection_password="${env.KC_DB_PASSWORD}"
connection_url="jdbc:postgresql://${env.KC_DB_URL_HOST}:${env.KC_DB_URL_PORT:5432}/${env.KC_DB_URL_DATABASE}${env.KC_DB_URL_PROPERTIES:}"
initialize_sql="CREATE SCHEMA IF NOT EXISTS ${env.KC_DB_SCHEMA:public}; CREATE TABLE IF NOT EXISTS ${env.KC_DB_SCHEMA:public}.JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, bind_addr varchar(200) NOT NULL, updated timestamp default current_timestamp, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
insert_single_sql="INSERT INTO ${env.KC_DB_SCHEMA:public}.JGROUPSPING (own_addr, cluster_name, bind_addr, updated, ping_data) values (?, ?, '${env.JGROUPS_DISCOVERY_EXTERNAL_IP:127.0.0.1}', NOW(), ?);"
delete_single_sql="DELETE FROM ${env.KC_DB_SCHEMA:public}.JGROUPSPING WHERE own_addr=? AND cluster_name=?;"
select_all_pingdata_sql="SELECT ping_data, own_addr, cluster_name FROM ${env.KC_DB_SCHEMA:public}.JGROUPSPING WHERE cluster_name=?"
info_writer_sleep_time="500"
remove_all_data_on_view_change="true"
stack.combine="REPLACE"
stack.position="MPING"
/>
</stack>
</jgroups>
<cache-container name="keycloak">
<!-- <transport lock-timeout="60000"/> -->
<transport lock-timeout="60000" stack="postgres-jdbc-ping-tcp"/>
<local-cache name="realms">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<local-cache name="users">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<distributed-cache name="sessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="authenticationSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="clientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="loginFailures" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<local-cache name="authorization">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<replicated-cache name="work">
<expiration lifespan="-1"/>
</replicated-cache>
<local-cache name="keys">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="3600000"/>
<memory max-count="1000"/>
</local-cache>
<distributed-cache name="actionTokens" owners="2">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="-1" lifespan="-1" interval="300000"/>
<memory max-count="-1"/>
</distributed-cache>
</cache-container>
</infinispan>
I tried changing the number of owners in the infinispan.xml but that did not work.
For anyone else struggling with this, the issue was because I had the different keycloak instances running on different ports.
In order for the auth Token to be validated, the issuing keycloak instance must have the same hostname and same port (same URL).See Doc
Once I setup a load balancer in front of keycloak instances and set it up as a proxy with my both my keycloak instances pointing to it, I was able to use their auth tokens interchangeably.