Monitoring Cron Jobs with Uptime Kuma Push Monitors (After Healthchecks Went Silent)

For weeks my Healthchecks dashboard was green, every job was checking in, and no alert could possibly have reached me. The process that sends alerts died every night a few seconds after it started, and nothing on screen said so.

A dead monitor that looks dead gets fixed. One that looks healthy just sits there while you trust it. So I rebuilt cron monitoring on Uptime Kuma, and this is how it works end to end, traps included.

TL;DR: one Kuma push monitor per job, fed by a small kuma_ping helper that reports status, a message, and run duration, and never fails the job. Alert time is interval + retries x retryInterval, sized per job. A push interval never equals the cron period. Maintenance windows get their members from a script, not by hand. And a VPS outside the house watches the box that runs Kuma.

The daemon nobody restarted

The Healthchecks container restarts every night at 23:08. On start, uWSGI launches manage.py sendalerts as an attach-daemon. Seconds later:

psycopg.OperationalError: [Errno -2] Name or service not known
daemon "./manage.py sendalerts --skip-checks" (pid: 14) annihilated

The hc-db hostname didn't resolve yet, because Docker was still bringing the network up (on unRAID that's the known shim-br0 DNS race). The daemon exits, uWSGI logs "annihilated", and never restarts it. The web workers connect to Postgres lazily, on the first request, and by then DNS works. So pings arrived, the page rendered, and the one piece that turns a missed ping into a message was dead. Every night.

That isn't really a Healthchecks design fault. It's a side process with no supervisor. In Kuma the check loop is the main process: if it dies, the container dies, and I notice.

One push monitor per job

Each job gets its own push monitor under a Cron Jobs group. The jobs live in a jobs.json registry, and provision.js reads it and creates any monitor that's missing. It's idempotent, skips names that exist, and has a DRY_RUN=1 mode. The push tokens never appear in my notes.

Kuma's push API is one GET:

GET https://kuma.example/api/push/<token>?status=up|down&msg=<text>&ping=<ms>

There's no /start or /log ping like Healthchecks has, and messages are cut at 400 characters. So the start of the run only records a timestamp, and the end of the run sends one beat. The migrated scripts all carry the same helper:

KUMA_PUSH_URL="https://kuma.example/api/push/<token>"
KUMA_START_TS=$(date +%s)
 
kuma_ping() {
    # $1 = message, $2 = status (up|down), default up. Never fails the caller.
    local status="${2:-up}"
    local ms=$(( ($(date +%s) - KUMA_START_TS) * 1000 ))
    curl -fsS -m 10 --retry 3 -G \
        --data-urlencode "status=${status}" \
        --data-urlencode "msg=${1:0:400}" \
        --data-urlencode "ping=${ms}" \
        "$KUMA_PUSH_URL" >/dev/null 2>&1 || true
}

A good run ends with an up beat and a short summary. A failure calls it with down and the reason, which lands in the alert text. The final beat carries the run duration in ping, so every job gets a duration graph for free. --data-urlencode keeps a message with spaces or & from breaking the URL. And || true is the part I care about most: a monitoring outage must never turn a good backup into a failed one. -m 10 --retry 3 keeps a slow Kuma from hanging the job.

The migration surfaced drift, too. Healthchecks expected the Glacier job monthly, but the unRAID scheduler runs it every three days, so it could never have alerted on a missed run inside a month. It expected the WSL backup every 2 hours; the crontab ran it every 6. Those monitors had been set from memory, not from the scheduler.

The gate before I retired Healthchecks: one real beat from every active push monitor. The last one due was Glacier, at 06:00 on 2026-08-24.

When a missed job actually pages me

A push monitor alerts after:

interval + retries x retryInterval

Inside the retry window Kuma shows the monitor as pending (yellow), not down. That's how every cron monitor is sized:

Alerts go to Telegram only, on every monitor. Email and Ntfy aren't attached anywhere. While a monitor stays down it re-alerts about every 30 minutes; for a daily push monitor that works out to a re-alert on every missed beat.

Two edges. Kuma rejects any interval above 2073600 seconds, which is 24 days, so a monthly job can't get a monthly interval. It gets a shorter interval and a long retryInterval to cover the gap. And a monitor for a job that only runs @reboot isn't a job check. My WSL workdir cleanup is one: its 48 h monitor really answers "has the PC rebooted in two days?". I inherited that one from the old setup, and I still have to decide whether to accept it or give the job a periodic run.

Never set the interval equal to the cron period

The NUC runs a heartbeat every minute: it reads 1 MB of /boot/bzmodules with iflag=direct, a real read of the USB boot flash rather than a cache hit, then pushes to Kuma. I first gave that monitor a 60 s interval, the same as the cron.

One night I got a NUC-down alert that lasted about an hour. The NUC was fine. The syslog had 59 lines of kuma push failed, and Kuma's heartbeat table had zero up beats for that hour.

The cause is an open upstream race in Kuma 2.5.3 (uptime-kuma#5357, fix in PR #7554, not merged). When Kuma's retry timer and an incoming push write the same stat_* row in the same second, the loser caches a broken row. Every later push in that stats bucket fails with HTTP 500 and SQLITE_CONSTRAINT. With interval equal to cron period, the two collide on a schedule.

The fix was 90 s. Another monitor had run at 90 s on a one-minute cron the whole time and never hit it. Now it's a rule for every push monitor fed by cron: never the same number as the cron period.

Maintenance windows go stale on their own

The nightly appdata backup stops containers from 03:00, so a 60-minute Kuma maintenance window covers it. Maintenance shows up as heartbeat status=3; a real failure inside the window is status=0.

I got the membership wrong twice. The first window held only 10 monitors from before the migration. Later it was missing every per-container monitor I'd added after building the window, plus a few app and push monitors: about 8 down and 8 up Telegram alerts every night, each doubled by its group monitor. The window hadn't broken. The fleet had grown and the list hadn't.

So I stopped keeping the list by hand. maintenance.js builds the windows, diffs by monitor id, and adds every Docker monitor whose container is in the backup's stop list by itself. A new container monitor needs no edit.

The opposite mistake is just as quiet. Postgres used to be in the window, but the backup never stops that container, so covering it would have hidden a real 3 a.m. outage. It came out. The unRAID web UI and SSH monitors stay armed on purpose: they're the only two that see a host crash where the containers keep running. Those exclusions are written down with their reasons, so nobody "fixes" them later. To prove a window works, I query a copy of kuma.db:

select m.name, h.time, h.status, h.msg from heartbeat h join monitor m on m.id=h.monitor_id
where h.time between '2026-08-21 23:00' and '2026-08-22 00:00' and h.status in (0,3);

A monitor that should be covered but logs status=0 inside the window isn't attached.

Watching the watcher

None of this helps when the NUC itself dies, because Kuma dies with it. So a small hub VPS outside the house runs a timer every minute with three checks:

1. Ping the NUC over WireGuard.
2. Fetch a public Kuma status page through Caddy that exposes only the flash heartbeat monitor.
3. Check that monitor's last heartbeat is up.

Three consecutive failures send a Telegram alert with the reason, repeated every 30 minutes while down, and one "back" message on recovery. Between them the checks cover power, kernel and WireGuard (ping), Caddy, Docker, the cache SSD and Kuma (status page), and a dropped USB flash or dead cron (heartbeat). Each run also pushes to its own Kuma monitor, so Kuma alerts if the watcher stops. There's no SSH or key in either direction, on purpose: a compromised hub must not reach the NUC.

It earned its place. When the boot flash dropped off the USB bus one night, the flash heartbeat went pending, then down at 23:58. The hub alert came at 23:59.