So right now we are trying to get a Bitnami Redis Sentinel cluster working, together with our Rails app + Sidekiq.
We tried different things, but it's not really clear to us, how we should specify the sentinels for Sidekiq (crutial part is here, that the sentinel nodes are READ ONLY, so we cannot use them for sidekiq, since job states get written).
Since on Kubernetes there are only 2 services available: "redis" and "redis-headless" (not sure how they differ?) - how can I specify the sentinels like this:
Sidekiq.configure_server do |config|
config.redis = {
url: "redis",
sentinels: [
{ host: "?", port: 26379 } # why do we have to specify it here seperately, since we should be able to get a unified answer via a service, or?
{ host: "?", port: 26379 }
{ host: "?", port: 26379 }
}
}
end
Would be nice if someone can shed some light on this. As far as I understood, the bitnami redis sentiel only returns the IP of the master and the application has to handle the corresponding writes to this master then (https://github.com/bitnami/charts/tree/master/bitnami/redis#master-replicas-with-sentinel) - but I really don't understand on how to do this with sidekiq?
Let's get started by clarifying the difference between a Headless Service and a Service.
A Service allows one to connect to one Pod, while a headless Service returns the list of available IP addresses from all the available pods, allowing to auto-discover.
A better detailed explanation by Marco Luksa has been published on SO here:
Each connection to the service is forwarded to one randomly selected backing pod. But what if the client needs to connect to all of those pods? What if the backing pods themselves need to each connect to all the other backing pods. Connecting through the service clearly isn’t the way to do this. What is?
For a client to connect to all pods, it needs to figure out the the IP of each individual pod. One option is to have the client call the Kubernetes API server and get the list of pods and their IP addresses through an API call, but because you should always strive to keep your apps Kubernetes-agnostic, using the API server isn’t ideal
Luckily, Kubernetes allows clients to discover pod IPs through DNS lookups. Usually, when you perform a DNS lookup for a service, the DNS server returns a single IP — the service’s cluster IP. But if you tell Kubernetes you don’t need a cluster IP for your service (you do this by setting the clusterIP field to None in the service specification ), the DNS server will return the pod IPs instead of the single service IP. Instead of returning a single DNS A record, the DNS server will return multiple A records for the service, each pointing to the IP of an individual pod backing the service at that moment. Clients can therefore do a simple DNS A record lookup and get the IPs of all the pods that are part of the service. The client can then use that information to connect to one, many, or all of them.
Setting the clusterIP field in a service spec to None makes the service headless, as Kubernetes won’t assign it a cluster IP through which clients could connect to the pods backing it.
"Kubernetes in Action" by Marco Luksa
As the Redis documentation say:
When using the Sentinel support you need to specify a list of sentinels to connect to. The list does not need to enumerate all your Sentinel instances, but a few so that if one is down the client will try the next one. The client is able to remember the last Sentinel that was able to reply correctly and will use it for the next requests.
So the idea is to give what you have, and if you scale up the redis pods, then you don't need to re-configure Sidekiq (or Rails if you're using Redis for caching).
Now you just need a way to fetch the IP addresses from the headless service in Ruby, and configure Redis client sentinels.
Fortunately, since Ruby 2.5.0, the Resolv class is available and can do that for you.
irb(main):007:0> Resolv.getaddresses "redis-headless"
=> ["172.16.105.95", "172.16.105.194", "172.16.9.197"]
So that you could do:
Sidekiq.configure_server do |config|
config.redis = {
# This `host` parameter is used by the Redis gem with the Redis command
# `get-master-addr-by-name` (See https://redis.io/topics/sentinel#obtaining-the-address-of-the-current-master)
# in order to retrieve the current Redis master IP address.
host: "mymaster",
sentinels: Resolv.getaddresses('redis-headless').map do |address|
{ host: address, port: 26379 }
end
}
end
That will create an Array of Hashes with the IP address as host: and 26379 as the port:.