Is it possible to create a Redis Cluster within Kubernetes using a Istio Search Mesh?

I am trying to configure a Redis cluster onto Kubernetes with an Istio Mesh installed. The Redis Cluster is able to be created without Istio and each Pods are auto-injected with an Istio Proxy (Envoy). However, with Istio installed and the Istio proxy attached to each Redis Pods, the Redis cluster is not able to "meet" correctly through the CLUSTER MEET command from the CLI.

For instance, I have Redis Pod A (slot 0 - 10919) and Redis Pod B (slot 10920 - 16383). This is the result after attempting a CLUSTER MEET command between them (cluster meet ClusterIPForRedisPodB 6379).

For Redis Pod A, the cluster info is updated and includes Redis Pod B:

On the contrary, for Redis Pod B, the cluster info is not updated and does not include Redis Pod A:

I am able to send curl and netcat responses between the two Pods for Port 16379 and 6379. In addition, Envoy appears to have these ports opened as well.

I have replicated your issue and found a solution to your problem.

Let me start by explaining what was the cause of your problem.

Redis gossip protocol works this way: when you type cluster meet <ip> <port> on redis1, redis1 opens a tcp connection to redis2. In normal case, when redis2 receives a connection, it accepts it, looks up source ip address of who's connecting and also opens a tcp connection to that address, so to redis1 in that case. (More on how gossip protocol works inredis can be found in redis documentation, or in this article)

Here comes the istio part. Istio by default configures envoy as a typical proxy and as you can read in istio documentation:

Istio be default is using REDIRECT proxing and as states in documentation:

This mode loses source IP addresses during redirection

This is our source of problem.

redis2 when receives a connection, it sees it as coming from localhost. Envoy has lost source IP address of redis1 and redis2 is now unable to open a connection back to redis1.

Now, we have some options:

1. you can try to change proxing mode to TPROXY (I tried it but couldn't make it work)
2. use redis built-in config variable

Lets take a look at the second option a bit closer because this is the one that worked for me. In redis.conf file you can find this section:

CLUSTER DOCKER/NAT support

In certain deployments, Redis Cluster nodes address discovery fails, because addresses are NAT-ted or because ports are forwarded (the typical case is Docker and other containers).

In order to make Redis Cluster working in such environments, a static configuration where each node knows its public address is needed. The following two options are used for this scope, and are:

• cluster-announce-ip
• cluster-announce-port
• cluster-announce-bus-port

Each instruct the node about its address, client port, and cluster message bus port. The information is then published in the header of the bus packets so that other nodes will be able to correctly map the address of the node publishing the information.

If the above options are not used, the normal Redis Cluster auto-detection will be used instead.

Note that when remapped, the bus port may not be at the fixed offset of clients port + 10000, so you can specify any port and bus-port depending on how they get remapped. If the bus-port is not set, a fixed offset of 10000 will be used as usually.

Example:

cluster-announce-ip 10.1.1.5
cluster-announce-port 6379
cluster-announce-bus-port 6380

We need to set cluster-announce-ip variable to redis's pod own ip address.

You can do it for example modifying redis-cluster ConfigMap like this (It's modified redis configmap from this article):

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
data:
  update-node.sh: |
      #!/bin/sh
      REDIS_NODES="/data/nodes.conf"
      sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
      cp /conf/redis.conf /redis.conf # <------HERE-----
      sed -i "s/MY_IP/${POD_IP}/" /redis.conf # <------HERE-----
      exec "$@"
  redis.conf: |+
      cluster-enabled yes
      cluster-require-full-coverage no
      cluster-node-timeout 15000
      cluster-config-file /data/nodes.conf
      cluster-migration-barrier 1
      appendonly yes
      protected-mode no
      cluster-announce-ip MY_IP # <------HERE-----

Also remember to change you container's command like this to point to the right redis.conf file:

command: ["/conf/update-node.sh", "redis-server", "/redis.conf"]

Every redis node will now advertise this address as it's own so the other redis nodes will now know how to connect with it.

Let me know if it helped.