A monitor has one job: to be believed when it speaks. Every false alarm spends some of that trust, and the real failure eventually arrives looking exactly like the ones you learned to dismiss.
What went wrong
Our daily backup check looked at whether the home server was reachable at the moment it ran. When the server moved off the office network and onto a connection across the internet, it was often briefly unreachable at that exact minute. One morning the check fired at 9:00 and the connection came back at 9:27. Twelve alerts like that had fired, four of them in five days, and none of them meant anything.
Alarm on how long, not on a single moment
We changed the check to ask how long something has been down, not whether it is down right now. It now alarms only after more than 26 hours: a day, plus slack. We tested it at 30 minutes, 25 hours, 27 hours, a week and "never seen": silent inside the window, loud outside it. Real errors, such as a sync folder reporting a failure, still alarm at once.
Check the destination, not the sender's log
Twice a log said everything was fine while nothing was arriving. One job's log cleared its own failure when a later run reported "no change". In another case the local state looked healthy while every upload was failing. The only reliable test is to look at the other end: is the newest copy actually there, and how old is it?
Let silence be the alarm
A checker that has stopped running sends no error. So ours reports every day to a small service elsewhere, and that service raises the alarm when a report doesn't arrive. A daily "all OK" message is also worth having: it proves the watcher is still watching.
Don't cry wolf about the network
- Retry, with pauses. Up to five attempts, waiting 15, 30, 45 and 60 seconds between them. Don't retry when the other side rejects the report outright; that needs a person.
- Wait for the network after waking. A laptop that wakes up runs its scheduled jobs before Wi-Fi is back. Ours now waits up to two and a half minutes for the connection, then checks anyway and says so.
- Warn for a blip, alarm for a stuck state. A brief outage is logged as a warning; only something that stays broken becomes an alert.
Watch for shrinking, too
A database that has been emptied still backs up "successfully". Our check compares the row count with the previous backup and flags a drop of more than 20%.
Say it once, remind, and say when it's over
Our mail server monitor alerts on the first failure, reminds every six hours while it lasts, and sends a message when things recover. You always know whether a problem is still open without digging.
For the backups themselves, see Backups you have actually restored.