The goal of the project is to wire up Vert.x applications(verticles) in a cluster with the help of Infinispan, where verticles share single replicated cache collection, which is also managed by Infinispan.
I can see the Infinispan's cluster management feature functioning fine when deployed on k8s. Below is the deployment information on kubernetes.
root@vertx-ollama-control-plane:/# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/backend-deployment-9dd7d994c-d6w5m 1/1 Running 1 (7m34s ago) 8m13s
pod/backend-deployment-9dd7d994c-d7ttq 1/1 Running 0 8m13s
pod/backend-deployment-9dd7d994c-zgjhx 1/1 Running 1 (7m34s ago) 8m13s
pod/frontend-deployment-6557dd4466-bf2gj 1/1 Running 0 8m13s
pod/frontend-deployment-6557dd4466-f6fqz 1/1 Running 0 8m13s
pod/ollama-858d4f8c8d-fggwb 1/1 Running 0 8m13s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/clustered-app ClusterIP None <none> 7800/TCP 8m13s
service/frontend LoadBalancer 10.96.229.141 <pending> 80:32375/TCP 8m13s
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h26m
service/ollama ClusterIP 10.96.54.50 <none> 11434/TCP 8m13s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/backend-deployment 3/3 3 3 8m13s
deployment.apps/frontend-deployment 2/2 2 2 8m13s
deployment.apps/ollama 1/1 1 1 8m13s
NAME DESIRED CURRENT READY AGE
replicaset.apps/backend-deployment-9dd7d994c 3 3 3 8m13s
replicaset.apps/frontend-deployment-6557dd4466 2 2 2 8m13s
replicaset.apps/ollama-858d4f8c8d 1 1 1 8m13s
And the exposed ip addresses of verticles through the headless service are here:
root@vertx-ollama-control-plane:/# kubectl get endpoints
NAME ENDPOINTS AGE
clustered-app 10.244.0.80:7800,10.244.0.81:7800,10.244.0.82:7800 + 2 more... 9m16s
frontend 10.244.0.83:8080,10.244.0.84:8080 9m16s
kubernetes 172.18.0.4:6443 3h27m
ollama 10.244.0.79:11434 9m16s
But the problem is that Pods do not share the cache of Infinispan (with the name of "embeddings"), rather have their own local collection. Here's some configuration and code I tried to manage the shared cache over cluster.
// Configure default cache manager
DefaultCacheManager cacheManager = new DefaultCacheManager(
new GlobalConfigurationBuilder()
.transport()
.defaultTransport()
.build()
);
clusterManager = new InfinispanClusterManager(cacheManager);
// Configure the cache for embeddings
Configuration cacheConfig = new ConfigurationBuilder().clustering()
.cacheMode(CacheMode.REPL_ASYNC)
.encoding()
.mediaType(MediaType.APPLICATION_OBJECT_TYPE)
.build();
...
if (cacheManager.cacheExists("embeddings")) {
logger.info(String.format("Cache %s exists with the hashcode of %d on %s node.",
"embeddings",
cacheManager.getCache("embeddings").hashCode(),
cacheManager.getNodeAddress())
);
collection = cacheManager.getCache("embeddings");
} else {
logger.info(String.format("Cache %s does not exist, a new cache is created on %s node.",
"embeddings",
cacheManager.getNodeAddress()
));
collection = cacheManager.createCache("embeddings", cacheConfig);
}
// ...
public static void main(String[] args) {
Vertx.clusteredVertx(new VertxOptions().setClusterManager(clusterManager))
.compose(v -> v.deployVerticle(new Main()))
.onFailure(Throwable::printStackTrace);
}
I am injecting the Jgroup Kubernetes xml config file as environment when deploying the pod like -Dvertx.jgroups.config=default-configs/default-jgroups-kubernetes.xml.
However, when storing embedding information in the Infinispan cache multiple times, I notice that each pod creates its own cache named 'embeddings.'
http POST :8080/embed prompt="Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels"
HTTP/1.1 200 OK
content-length: 105
Embedding entry stored with key: 451439790
From: backend-deployment-9dd7d994c-zgjhx (Collection Size: 1)
...**(logs from the Pod A)**
Dec 14, 2024 4:10:26 AM cynicdog.io.api.OllamaAPI
INFO: **Cache embeddings does not exist**, a new cache is created on backend-deployment-9dd7d994c-d6w5m-37175 node
http POST :8080/embed prompt="Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels"
HTTP/1.1 200 OK
content-length: 105
Embedding entry stored with key: 451439790
From: backend-deployment-9dd7d994c-d6w5m (Collection Size: 1)
...**(logs from the Pod B) **
INFO: Model mxbai-embed-large:latest pulled.
Dec 14, 2024 4:10:26 AM cynicdog.io.api.OllamaAPI
INFO: **Cache embeddings does not exist**, a new cache is created on backend-deployment-9dd7d994c-d6w5m-37175 node.
Is there anything I'm missing? I'm attaching the project GitHub repository if needed for detailed inspection :(
https://github.com/CynicDog/Vertx-Kubernetes-Integration/tree/main/clustered-embedding-stores
When you create a cluster manager using a custom cache manager, this cache manager must be configured with the required caches:
<cache-container default-cache="distributed-cache">
<distributed-cache name="distributed-cache"/>
<replicated-cache name="__vertx.subs"/>
<replicated-cache name="__vertx.haInfo"/>
<replicated-cache name="__vertx.nodeInfo"/>
<distributed-cache-configuration name="__vertx.distributed.cache.configuration"/>
</cache-container>
This is XML configuration but you should be able to do the same programmatically.
Secondly, you must follow the steps for configuring Infinispan on K8S
But note that, as indicated in the comments, the vertx.jgroups.config is ignored when the cluster manager is created with a custom cache manager.
So you must configure the JGroups programmatically.