How can I get dockerized nginx to failover quickly when one of the upstream containers goes down? Failover seems instantaneous in a non-docker environment, but several requests timeout when dockerized.
I'm using nginx as a load balancer/proxy on a single node/vm in front of two instances of the same node app. I pictured new version deployments to go as follows (often called blue-green deployments):
However, on nginx doesn't seamlessly do #4. When I take down one of the upstream containers for version bump, nginx will timeout several requests (in round-robin fashion) as it figures out one of the containers is down. I haven't had this happen in a non-docker environment.
Here's my docker cloud stack file:
load-balancer:
image: 'foo/load-balancer:latest'
links:
- node-blue
- node-green
ports:
- '80:80'
node-blue:
image: 'foo/node-app:latest'
node-green:
image: 'foo/node-app:latest'
nginx.conf looks like:
events {
worker_connections 1024;
}
http {
gzip on;
upstream app {
server node-green;
server node-blue;
}
server {
listen 80;
server_name app.local;
location / {
proxy_pass http://app;
}
}
}
After a bunch of wrestling, I think I came out with a reasonable solution. Swapped out a custom nginx container for dockercloud/haproxy. Apparently the load balancer needs to be able to listen to docker cloud events, and adjust accordingly.
Stack file now looks like:
load-balancer:
image: 'dockercloud/haproxy:1.5.3'
restart: always
roles:
- global
links:
- node-blue
ports:
- '80:80'
node-blue:
image: 'foo/node-app:latest'
environment:
- VIRTUAL_HOST=app.local
node-green:
image: 'foo/node-app:latest'
environment:
- VIRTUAL_HOST=app.local
And when I want to deploy a new version, I just run the following script:
docker-cloud service set --link node-green:node-green load-balancer
docker-cloud service redeploy node-blue --sync
docker-cloud service set --link node-blue:node-blue load-balancer
docker-cloud service redeploy node-green --sync