microservicesmesosmesosphereconsulconsul-template

Consul deregister 'failing' services


I have consul running on Consul v0.5.2 version & services running in Mesos. Services keep moving from 1 server to another.

Is there way to deregister services in consul that are in 'failing' state? I am able to get the list of services in failing state using this curl

curl http://localhost:8500/v1/health/state/critical

Issue that we are seeing is over a period of time in consul UI we have stale data & making the whole UI unusable


Solution

  • Consul by default do not deregister unhealthy services instead marks them as critical. From Consul 0.7 there is special option (deregister_critical_service_after) that allows you to define time after unhealthy service will be deregstered

    From Consul 0.7 Changelog

    Automatic Service Deregistration: Added a new deregister_critical_service_after timeout field for health checks which will cause the service associated with that check to get deregistered if the check is critical for longer than the timeout. This is useful for cleanup of health checks registered natively by applications, or in other situations where services may not always be cleanly shutdown. GH-679

    If you are usign Marathon then you can consider using allegro/marathon-consul it will deregister task when its dead