dockerdocker-composedocker-swarmdocker-swarm-mode

Rollback whole swarm when one service fails


Is it possible to rollback another services when only one fails on update with docker stack deploy?

For example I have serv1, serv2, serv3. I run docker stack deploy, serv1 and serv2 update without any problems, but then serv3 fails on healthcheck. Now I want serv1, serv2 and serv3 to rollback to previous version.

Is that possible in docker swarm mode?


Solution

  • You can add pinging each other to each service's health check.

    So, service 1 health check - ensures service 1 is running, and both service 2 and service 3 is reachable. For service 2 - ping 1st and 3rd services For service 3 - ping 2nd and 3rd.

    As result, if any of services fails, its restarted, if it fails to ping any of dependent ones, its restarted too.

    I encountered analogous issue, so, we added 4th service - the global healthchecker one, that has docker socket mounted from host, and it pinged all services. It was custom and, quite ugly, nodejs script. It pinged/checked each of services it was watching, and if any of them failed, it send docker service update --force serv1 or something like this to docker socket bound from host machine.