How should I script daemontools superviser?
I've started to use D. J. Bernsteins' daemontools to keep some processes running and is working fine. But now I need to monitor a couple of additional conditions on those processes and I've failed to find good info on how to do that.
My scenario is that I have some processes running for a web app (pharo smalltalk virtual machines) and they respond http, each one in their port (that's for the for loadbalance). I would like to somehow ping those to verify that they are not only running but responding to http requests. If they don't respond in a certain way to a request for more than 30s they should be treated as crashed and simply be restarted.
Is this even possible with daemontools? if so, how should I write this script and where should I place it? or where's the documentation on this?
The simplest solution is to create another daemontool task with a script that sleeps for 30 seconds and then tests for the presence of the service (using wget or curl for example). If the service doesn't respond timely you can restart the service (svc -t yourapp) and/or send a notification. The run-script of the new service could look as simple as this:
#!/bin/sh
sleep 30
if ! wget --quiet --timeout=5 --delete-after "http://yourapp.com/" ; then
svc -t /etc/service/yourapp
fi
I've also made good experience with tools like Munin. Again you need to provide a script that provides information about the state of your image. If you setup your images with a REST service you can even provide really interesting metrics such as active sessions, inactive session, gc parameters, memory consumption, database statistics, ... The tool then draws nice graphs over time and lets you specify boundaries to get notified when things behave badly.