I'm trying to use KEDA to auto-scale a Celery worker pod based on the current length of a RabbitMQ message queue. The worker runs as expected, however, it is not scaling under load.
For reference, I'm using KEDA v2.7.1.
Logs from keda-operator pod:
1.6569289126033094e+09 ERROR controller.scaledobject Reconciler error {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "celery-worker-scaler", "namespace": "default", "error": "error establishing rabbitmq connection: dial tcp: lookup rabbitmq on 10.96.0.10:53: server misbehaving"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227
Clearly there is some issue with connecting to the rabbitmq pod
keda.yaml:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: celery-worker-scaler
namespace: default
spec:
scaleTargetRef:
name: celery-worker
pollingInterval: 3
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: rabbitmq
metadata:
host: amqp://guest:guest@rabbitmq:5672
protocol: amqp
queueName: celery
mode: QueueLength
value: "5"
rabbitmq.yaml:
# deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: rabbitmq
labels:
app: rabbitmq
spec:
replicas: 1
selector:
matchLabels:
app: rabbitmq
template:
metadata:
labels:
app: rabbitmq
spec:
containers:
- name: rabbitmq
image: rabbitmq:management
env:
- name: RABBITMQ_DEFAULT_USER
valueFrom:
secretKeyRef:
name: secret
key: rabbitmq-user
- name: RABBITMQ_DEFAULT_PASS
valueFrom:
secretKeyRef:
name: secret
key: rabbitmq-password
ports:
- containerPort: 15672
name: management
- containerPort: 5672
name: rabbitmq
---
# service
apiVersion: v1
kind: Service
metadata:
name: rabbitmq
labels:
app: rabbitmq
spec:
selector:
app: rabbitmq
ports:
- port: 5672
name: rabbitmq
targetPort: rabbitmq
protocol: TCP
- port: 15672
name: rabbitmq-mgmt
targetPort: management
The keda.yaml file is applied in the keda namespace (even though I wrote default at the top). Since my rabbitmq pod is in the default namespace, I had to set the host as follows: amqp://guest:guest@rabbitmq.default:5672