Monitoring system and service uptime with heartbeats?

[IridiuM]

How do all the 1337 developers out there monitor their system and service uptime?

I worked on https://heartbeat.sh which is a free service that does just that (complete with dark mode ), it may be useful for anybody looking to monitor the execution of scripts, cronjobs, server uptime, backups, processes etc. - Hell just about anything... (did the gate motor close?)

Shell script: https://github.com/heartbeat-sh/heartbeat.sh
Python client (PIP): https://github.com/heartbeat-sh/heartbeat.py
JS client (NPM): https://github.com/heartbeat-sh/heartbeat.js

Let me know what you think or help me make it better if you're keen to jump in an make use of it!

^_0

 

[IridiuM]

Hi Eric (on behalf of ariedebok - who is banned for no reason -> plz someone unban him)

Thanks for your feedback! I can see that you gave this some serious thought, and I will definitely implement some of your suggestions. Adding a notification system is my next development goal.

We use the heartbeat system for our in house monitoring at Direct Debit and SnapBill. We created heartbeat.sh, because the system works well for us, and we want to make it available as a service to the public. For now it is a "beta" version to see the reaction of the public, but the idea is to make it permanent and to grow it according to the feedback we receive.

We host the heartbeat.sh through AWS, we don't have a self-hosted version available currently. Redundancy-wise we have 2 app servers. The heartbeats themselves are stored in a Redis instance with redundancy spread over 3 nodes. We store user details in a Postgres DB on Amazon's RDS service. We make daily snapshots of the DB.

As for the information we keep: we store heartbeat details (timestamp of the last beat, configured timeouts, name, subdomain) and login details for the users that registered an account. The user passwords are hashed in our db, and every password is hashed with a randomly generated salt (as opposed to using the same salt for every password). Our hashing algorithm is cryptographically secure.

I've started to write some how-to's on our documentation page. For most cases, it is good enough to just send a heartbeat at the end of every script run, making the timeout a bit longer than the frequency of the script.

The approach of sending start and end heartbeats is good for mission critical tasks, where you want to be notified early. I would keep one heartbeat, and just swop out the timeouts. If you expect a backup to take 30 minutes, send a heartbeat with a 30 minute timeout at the start. Once the backup is done, send a heartbeat with the same name, but a 24 hour timeout.

You can also send a heartbeat with a 0 second timeout in case something went wrong. And then delete the heartbeat once you fixed the issue.

Our alert mechanism is still under development. Fow now, polling the API is best. If you know your way around the shell, I would recommend polling https://{subdomain}.heartbeat.sh/heartbeats/text. As that would give a response that can easily be parsed with awk. If you prefer to work with JSON, you could just use /heartbeats/json instead.

For our in house monitoring, we hooked up a robot(yes, a traffic light) to a raspberry pi. The pi polls our API, and changes the robot's colour according to the status.

Thanks again for your feedback!
Arie, heartbeat.sh developer

 

[IridiuM]

The robot in action. We're all good!

 

[Anthro]

Someone call the Govenment, this man stole a light from a random intersection !

Spoiler

(What Im actually asking is.. where did you "buy" that from ?

 

[Gnome]

Are you new to software development?

Heartbeat the way you phrase it sounded much more like metrics to me.

Anyway the most common way to monitor is metrics and alarming. The alarm can either be to send an email or push a notification. In my company we have a paging app installed on our phones and it sets off an alarm for the engineering to check in and fix the problem.

If you are "anti cloud" then you can build it yourself and get something less useful and less reliable for more money I guess.

Traffic lights are great if you only care about having your app running when people are in the office. But having worked in companies where 99.99% uptime is a requirement, I see the traffic light thing as a bit of an amateur hour.

"Heartbeat", the phrase, is more commonly applied pattern in a leader election based multi server application, for example using paxos or a number of other leader election strategies. The leader heartbeats so that another agent can take over if it dies.

 

[IridiuM]

You sound like you're 12 years old.
You have no idea what the product is so rather stop. There are links in the OP, you didn't even have to Google.

This is actually pretty innovative. A central portal you can use for monitoring any service or script uptime where you don't need access inwards to the service or special routines to do that. Your system merely needs Https out and can transmit a 'heartbeat' using any number of existing libraries, scripts or code. You can send heartbeats to mark the start and end of a process, or just to announce "I'm alive and well". You set a timeout on the heartbeat and if the process takes longer than expected, you can receive alerts any which way you like to announce there's an issue. Perfect for retrofitting legacy systems or a quick solution with very little effort on a small project.

It's a great tool for resolving some false negatives in your monitoring since you push the heartbeat from the service side.

The traffic light? It's a gimmick that's a bit of fun. But fun seems like your middle name so you should have got that LMAO.

What Eric said ...