Minitor: A Minimal Monitoring Tool

If a service isn’t monitored, is it really running? The age old question. Since I started my voyage into self hosting, it had been mostly smooth sailing. That is, until the other day when, after about a year, I had my first server outage.

The importance of knowing

It was a gradually intensifying experience. I noticed that my email send resulted in an error and then saw some new calendar events stop syncing. I thought maybe some services were down or slow. After an hour or so, I got to my computer and couldn’t even open a remote shell. That’s when I realized I was in trouble!

If I had some kind of notification, I would have been able to jump right in and begin restoring my backups immediately. My server is running Cloudron to manage my services, and Cloudron has notifications built in for services it’s running. However, if the server goes down there’s nothing to send the notification email.

Since I’m hosting everything, from my email to calendar events, it is decidedly important to know if something goes down.

This should be easy!

At work we host our services using Paasta, which includes alerting through Sensu, and it's amazing! So my first thought was, "I should check out Sensu!", and my next thought thought was "Wow, this is way overkill!" There just has to be a simple way to do this. All I really needed was to run a command periodically, and do something if it fails. After reseraching a few other tools, I decided it was time to try my hand at writing one. Afterall, it's not a crypto library!

There just has to be a simple way to do this.

With simplicity in mind, I decided I wanted the tool to do as little as possible and take advantage to as many existing technoligies as I could. There are already great *nix tools for checking if ports are open, if you can get responses back from services, and even inspecting Docker containers. There are also great tools for sending emails, SMS notifications, push messages, or any number of potential downstream actions. So why reinvent the wheel? This tool could just delegate the "complicated stuff" to other processes.

Minitor: Giving birth to a monster

Leaving so much up to other processes really reduces the complexity of the application. I was able to get a simple prototype with hard coded checks in a few minutes. After that, I prototyped the config format I'd like to use and got to work on the real deal. A little under 4 nights of hacking later, Minitor (Github mirror) (Docker Hub) was born!

Making it configurable

I wanted some way to set the check command and reuse alert commands. It would also be very useful to have some kind of backoff and alert after settings to prevent a storm of notifications for a short outage. I ended up with this config:

monitors:
  - name: My Blog
    command: [ 'curl', '-s', '-o', '/dev/null', 'https://blog.iamthefij.com' ]
    alerts: [ log, mailgun ]
    check_interval: 30
    alert_after: 3
    alert_every: -1 # Defaults to -1 for exponential backoff

alerts:
  mailgun:
    command: >
      curl -s -X POST
      -F subject="Alert! {monitor_name} failed"
      -F from="Minitor <minitor@minitor.mon>"
      -F to=me@minitor.mon
      -F text="Our monitor failed"
      https://api.mailgun.net/v3/minitor.mon/messages
      -u "api:${MAILGUN_API_KEY}"

This would allow me to check every so many seconds, control when to alert, and then peform some action (sending an email, posting to a URL, attempt to restart a service) until recovery.

Why you should use it

Minitor is a drop dead simple service for checking things and doing things when they fail. No mucking around having to find or write plugins. A new proccess is your plugin adapter! Use any command you can find or a custom shell script as your plugin.

Minitor is simple to configure. Using an easily readible yaml file, you can set commands to be executed with or without a shell, much like how Docker executes them. You can configure check intervals, and backoffs globally and per monitor. And you can delegate alerts to a number of other commands. This allows you to send emails, SMS

Minitor is easy to run. It's a Python package and installable with pip. Intall with pip install minitor and run with minitor. No incantations or arguments to remember. Just start it and it will read it's config file and start monitoring.

The future

I already said Minitor is simple, right? Well, one of the reasons is it does only a few things. The entire application is about 250 lines of Python. This means that there are some things it doesn't do yet, and other things it will never do.

Minitor will never be as robust as Sensu or Nagios. It will not have complicated installable plugins, but will continue to integrate through calling other processes. I'm documenting potential features and requests on the repo, but not all will be implemented. The goal is to make sure Minitor stays minimal as well as useful.

PS. Looking for a cool (libre) logo! Let me know on the repo if you have any ideas.