Server monitoring that talks to me: a Telegram alerting bot for SSH logins, service health and tampered repositories

Why not a SaaS

Because the interesting signals are on the box: auth.log, authorized_keys, systemctl, df. An agent that ships all of that to a vendor was more surface than I wanted, and every hosted tool I tried buried the one alert that mattered under dashboards. The whole point was fewer messages, each one worth reading.

Generation 1 — Bash and cron, in production

Three scripts, three cron entries, and curl to the Telegram sendMessage endpoint. Send-only by design: nothing listens on the network.

What monitor.sh alerts on:

• An SSH login from an address never seen before. The first-seen list is strict, and the message says whether the address falls inside my usual ranges or not — a login from my own ISP at 2 a.m. and a login from another continent are not the same alert.
• A login with a revoked key. Every key I have ever rotated out is kept by fingerprint; if one is used successfully, that is a critical alert, because it means a copy exists somewhere it should not.
• Drift in authorized_keys — any key that is not on the approved list.
• A second UID-0 account. A classic persistence trick, and a one-line check.
• A remote host that stops answering, reported once per outage rather than every two minutes.

Design details that keep it quiet:

• An 8-minute look-back window so a run never re-reads history it has already reported.
• De-duplication by hashed signature: each alert's identity is a truncated SHA-256 kept in a state file (capped at 2,000 entries), so a flapping condition is reported once.
• Deploy keys from CI are expected — but only from CI. Logins with the deploy key are suppressed only when the source address is inside GitHub's published Actions ranges, refreshed daily from GitHub's meta API. The same key used from anywhere else alerts as possible key theft.
• Every alert is also appended to a local audit log, so the Telegram history is not the only record.

The repository watcher

This part exists because of an incident I wrote up separately: malware injected into my repositories by rewriting legitimate commits. After cleaning up, I wanted to know within half an hour if it happened again.

repo-guard.sh is read-only and works without any GitHub token for public repositories:

1. It records the SHA of every branch of every watched repository with git ls-remote. A SHA that moves backwards, a branch that disappears or a force-push shows up as a diff against the last run.
2. Changed repositories are shallow-cloned and scanned by content, never by file name: the obfuscated payload's signature strings, .woff2 "fonts" whose first bytes are not wOF2, VS Code tasks.json files with a folderOpen trigger, abnormally large or single-line build config files, and commits whose committer differs from their author.
3. Once a day it does a full scan instead of a shallow one.

Generation 2 — the Python rewrite

The Bash version is tied to my boxes. vps-sentinel is the portable one: a single Python file, standard library only, configured entirely through environment variables, shipped with a hardened systemd unit (dedicated user, NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, one writable state directory).

It adds what the cron scripts cannot do:

• Commands. /status, /logins and /help, answered only for the allow-listed chat id, over long-polling getUpdates — still no inbound port.
• Rotation-aware log tailing. auth.log is followed by inode and offset, and the first pass is swallowed so a restart never replays history.
• Hysteresis everywhere. Disk alerts at 85 % and re-arm five points lower; load alerts above 2 × the core count and re-arm at 0.7 ×; a service that goes down is reported once, and its recovery is reported with the downtime.
• Failed-auth bursts (20 in five minutes by default) as a separate signal from successful logins.
• Atomic state writes (temp file + os.replace) and clean SIGTERM handling, because a monitor that corrupts its own state on a reboot is worse than none.

What I learned running it

Most alerting systems fail by being ignored. Most of the rules above exist because of an alert that was once noise: the flapping service, the deploy key that logged in on every deploy, the daily "all good" that stopped being read. The current version sends me a handful of messages a week, which is the only reason I still read them.