Monitoring dead-man (Healthchecks.io)¶

External heartbeat for total Prometheus / Alertmanager loss. A systemd timer on infra-services curls Healthchecks.io every five minutes. That path does not go through Alertmanager. When pings stop for longer than Period + Grace, Healthchecks.io pushes to a dedicated ntfy topic on your phone.

Architecture¶

homelab-monitoring-deadman.timer (every 5m on infra-services)
  → curl -fsS https://hc-ping.com/<uuid>
  → Healthchecks.io check stays UP while pings arrive
  → miss (Period + Grace) → Healthchecks ntfy integration
  → https://ntfy.sh/homelab-monitoring-deadman-<4hex>
  → phone notification (DOWN / UP)

flowchart LR
  T[deadman timer] -->|curl ping| HC[Healthchecks.io]
  HC -->|still pinging| Up[check UP]
  HC -->|miss Period+Grace| N[ntfy topic]
  N --> Phone[phone app]
  AM[Alertmanager] -.->|not on this path| N

Complementary Prom rules (AlertmanagerUnavailable, GrafanaUnavailable, …) cover partial failures while Prometheus can still scrape and notify. They are not a substitute for this dead-man.

Path	Who delivers to your phone
This dead-man	Healthchecks.io → ntfy (integration in HC UI)
Patch critical / capacity	Homelab curl / Alertmanager → ntfy
Stage 7 pilot notify	OneUptime adapter → separate pilot topics

Ansible only installs the timer and writes /etc/homelab/monitoring-deadman.env with healthchecks_ping_url. It does not post to ntfy.

Prerequisites¶

Healthchecks.io account (free plan is enough for one check).
Phone with the ntfy app (Android / iOS), or any client that can subscribe to https://ntfy.sh/<topic>.
Age key available where you encrypt SOPS (/etc/homelab/age-key.txt on infra-services, or your operator key).
Repo checkout with write access so you can commit ciphertext.

1. Pick a dedicated ntfy topic¶

Use a new topic. Do not reuse:

Existing topic	Purpose
`homelab-patch-critical-b4e9`	OS patching critical only
`homelab-whrrr-capacity-7f2a`	Synology volume capacity

Suggested name pattern (match the SOPS example):

homelab-monitoring-deadman-<4hex>

Example: homelab-monitoring-deadman-a1f3.

Invent four hex characters (or more) so the topic is hard to guess.
Write the full topic name down; you will use it in Healthchecks and on the phone. Optionally record it in ntfy_deadman_topic inside the SOPS file for documentation (Ansible does not send to that topic).

Install the ntfy app from ntfy.sh/app.
Subscribe to topic → enter exactly homelab-monitoring-deadman-<4hex> (server left as public ntfy.sh unless you self-host).
Optional but recommended for a dead-man:
Raise priority / “priority high” for this subscription.
Enable override Do Not Disturb / insistent alerting if your phone supports it — this alert means the monitoring plane may be dark.

Prove the topic works (before Healthchecks)¶

From any machine (Windows PowerShell or bash):

curl -d "homelab monitoring dead-man ntfy smoke" \
  -H "Title: Dead-man ntfy test" \
  -H "Priority: high" \
  -H "Tags: warning,skull" \
  "https://ntfy.sh/homelab-monitoring-deadman-<4hex>"

Expect a push on the phone within a few seconds. If nothing arrives:

Topic spelling mismatch (most common).
App subscribed to a different server than ntfy.sh.
Phone battery / notification permission blocking the app.
Corporate network blocking ntfy.sh (try mobile data).

Do not continue until this curl test lights up the phone.

2. Create the Healthchecks.io check¶

Sign in at healthchecks.io.
Add Check (or Project → Checks → Add).
Set timing to match the timer:

Field	Value	Why
Period	5 minutes	Matches `OnUnitActiveSec=5min` on the timer
Grace	5 minutes	Miss window before DOWN (Period + Grace ≈ 10 minutes)

Name it something obvious, e.g. homelab monitoring dead-man / infra-services prometheus plane.
Save, then open the check and copy the Ping URL:

https://hc-ping.com/<uuid>

That UUID is a secret. Treat it like a password — it goes into SOPS, not into chat or screenshots if you can avoid it.

Optional: ping once from your laptop¶

curl -fsS -o /dev/null -w "%{http_code}\n" "https://hc-ping.com/<uuid>"

Expect 200 and the check flipping to a fresh last-ping in the HC UI. This does not enable the host timer yet; it only proves the URL.

3. Wire Healthchecks → ntfy (the important part)¶

Healthchecks owns delivery. You are configuring an integration on the check (or project), not Alertmanager and not Ansible.

In Healthchecks, open the check (or Project → Integrations).
Add / enable the ntfy integration (native integration — not a generic webhook unless you prefer that).
Topic / URL fields — use the same topic you subscribed to:

Typical field	Value
Topic or URL	`homelab-monitoring-deadman-<4hex>` or full `https://ntfy.sh/homelab-monitoring-deadman-<4hex>`
Priority	high / max if offered
Token	leave empty for a public topic on ntfy.sh

Assign the integration to this dead-man check (if the UI is project-wide, restrict or confirm it fires for this check’s DOWN/UP events).
Use Healthchecks’ Test / send test notification if available. Otherwise you will prove end-to-end in the Miss drill below.

What you should see on the phone later:

DOWN (or equivalent) when pings stop past Grace.
UP / recovered when pings resume.

If curl to ntfy works but HC never notifies:

Integration not attached to this check.
Wrong topic in the integration.
Check still receiving pings (timer still running during a “miss” test).
HC free-plan / project limits (unlikely for one check).

4. Encrypt the ping URL (SOPS)¶

File (only this one):

infra/ansible/inventory/group_vars/patch_controller/monitoring-deadman.sops.yaml

Example template (keep in sync when keys change):

infra/ansible/inventory/group_vars/patch_controller/monitoring-deadman.sops.yaml.example

Only healthchecks_ping_url is required for the timer. The ntfy_* keys in the example are documentation / smoke reference — Healthchecks still owns ntfy.

On a machine that can decrypt with the operator or host age key (often infra-services):

cd /path/to/homelab
cd infra/ansible/inventory/group_vars/patch_controller
cp monitoring-deadman.sops.yaml.example monitoring-deadman.sops.yaml

Edit plaintext monitoring-deadman.sops.yaml:

healthchecks_ping_url: "https://hc-ping.com/<uuid>"

ntfy_server: "https://ntfy.sh"
ntfy_deadman_topic: "homelab-monitoring-deadman-<4hex>"

Encrypt in place (do not commit plaintext):

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -e -i monitoring-deadman.sops.yaml

On Windows with WSL or a remote shell, same command once the age key path is correct. Confirm the file is ciphertext (sops: header / encrypted values) before git add.

git add infra/ansible/inventory/group_vars/patch_controller/monitoring-deadman.sops.yaml
git commit -m "Encrypt monitoring dead-man Healthchecks ping URL."
git push

Merge to main (PR or direct per your workflow). Hosts only apply after origin/main updates.

If the file already exists (ciphertext present), edit in place instead of cp:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops monitoring-deadman.sops.yaml
# set healthchecks_ping_url to the new Ping URL, save, exit

5. Apply on infra-services¶

After merge, wait for ansible-pull or run with the role tag:

# on infra-services, from the pull workdir or a checkout of main
sudo ansible-playbook ... --tags monitoring-deadman

The role writes /etc/homelab/monitoring-deadman.env (mode 0600) and enables homelab-monitoring-deadman.timer when the ping URL is non-empty. If SOPS is missing or the URL is still REPLACE_…, the timer stays disabled and Ansible logs a warning pointing at this runbook.

6. Verify the happy path¶

systemctl status homelab-monitoring-deadman.timer
systemctl list-timers homelab-monitoring-deadman.timer
sudo systemctl start homelab-monitoring-deadman.service
journalctl -u homelab-monitoring-deadman.service -n 20 --no-pager

Expect:

Timer enabled / active.
Oneshot exit 0 (curl succeeded).
Healthchecks UI shows a fresh ping within a minute.

Repo smoke helper (on the host):

bash scripts/test-monitoring-deadman.sh
bash scripts/test-monitoring-deadman.sh --live   # starts oneshot when configured

--live only proves the timer env and a ping; it does not exercise ntfy. Use the miss drill for ntfy.

Miss drill¶

Proves Healthchecks → ntfy without taking down Prometheus.

sudo systemctl stop homelab-monitoring-deadman.timer

Wait about Period + Grace (~10 minutes with 5m + 5m).
Phone should get a DOWN (or “check is down”) from Healthchecks via ntfy.
Healthchecks UI should show the check DOWN.

Recover:

sudo systemctl start homelab-monitoring-deadman.timer
sudo systemctl start homelab-monitoring-deadman.service

Healthchecks should flip UP after the next successful ping.
Phone should get an UP / recovered notification (if the integration sends recoveries — confirm in HC integration settings if UP is silent).

If DOWN never arrives: re-check §3 and that the timer is actually stopped (systemctl is-active should not be active for the timer).

Troubleshooting¶

Symptom	Likely cause	What to check
Timer disabled / no env file	SOPS missing or empty URL	Decrypt path; `healthchecks_ping_url`; role warning in ansible-pull log
Oneshot fails	Bad URL / network	`journalctl -u homelab-monitoring-deadman.service`; curl the ping URL manually
HC never gets pings	Timer not running or wrong host	Confirm role on infra-services; `list-timers`
Curl to ntfy works, miss drill silent	HC integration wrong	Topic spelling; integration attached to this check; wait full Grace
Patch/capacity alerts on this topic	Wrong topic reuse	Create a new topic; do not share patch-critical
Phone silent for everything	App / OS	Notification permission; DND; try mobile data

Synology capacity ntfy — same public ntfy.sh pattern, different topic
Coordinated OS patching — patch-critical topic homelab-patch-critical-b4e9
Example secrets: monitoring-deadman.sops.yaml.example
Role: infra/ansible/roles/monitoring-deadman
Smoke: scripts/test-monitoring-deadman.sh