I'm looking to create a monitor that will be in charge of keeping track of a bunch of different VM's (almost always ubuntu based). I plan on testing by sending requests, as well as testing if certain services are running on the specific VM's (this might need some sort of ssh component?). This data will be sent via a REST API to a cachethq instance.
Some of the services I will be monitoring include: Jira, Confluence, Jenkins (and slave nodes), as well as a few other Atlassian suite products.
I was thinking of writing something in python that would do this. Does anyone have any other suggestions that would be better?
thanks
Let's say you set up your monitoring server that does all the monitoring you described. And then your monitoring server goes offline. How will you find out?
Getting this recursive monitoring problem solved will take you more time than you probably estimated which is why it almost never makes sense to build one's own monitoring solution.
I'd use an inexpensive flexible uptime monitor (I'm partial towards https://wdt.io), set up cron jobs on your VMs that test your services and then curl a unique URL at the uptime monitor. If the service is down, the curl won't happen and the monitor will alert you.
Those services with a web gui, you can have monitored directly with the uptime monitor by having it ping the corresponding URL on a regular basis.