Monitoring dead-man (Healthchecks.io)

Independent host/timer/outbound-network heartbeat, not a detector of total Prometheus / Alertmanager loss. A systemd timer on infra-services curls Healthchecks.io every five minutes without checking either monitoring process. A healthy host can keep pinging while both Prometheus and Alertmanager are stopped. An UP check is evidence of recent ping receipt, not alert evaluation, receiver health or operator delivery.

The path does not go through Alertmanager. When pings stop for longer than Period + Grace, the configured Healthchecks.io integration should notify the dedicated ntfy topic. Delivery and phone receipt require separate acceptance. The setup and miss-drill commands below are owner-operated live procedures, not source-validation steps; they can send notifications or change host state.

Architecture

homelab-monitoring-deadman.timer (every 5m on infra-services)
  → curl -fsS https://hc-ping.com/<uuid>
  → Healthchecks.io check stays UP while pings arrive
  → miss (Period + Grace) → Healthchecks ntfy integration
  → https://ntfy.sh/homelab-monitoring-deadman-<4hex>
  → phone notification (DOWN / UP)

flowchart LR
  T[deadman timer] -->|curl ping| HC[Healthchecks.io]
  HC -->|still pinging| Up[check UP]
  HC -->|miss Period+Grace| N[ntfy topic]
  N --> Phone[phone app]
  AM[Alertmanager] -.->|not on this path| N

Complementary Prometheus rules (AlertmanagerUnavailable, GrafanaUnavailable, …) can observe partial failures only while evaluation works. Delivery still needs a functioning route/receiver: an alert about a failed Alertmanager cannot rely on that same failed Alertmanager to notify. Neither these rules nor this host heartbeat prove total monitoring-plane loss detection.

Coverage and acceptance boundaries

The unit template has only After/Wants=network-online.target and a bounded curl ExecStart; there is no Docker, Prometheus, Alertmanager or rule-evaluation probe. Keep that independent host signal. A separate monitoring-chain watchdog or a separately designed health-gated signal would need owner review and failure tests before claiming monitoring-plane coverage; neither is implemented by this timer.

Failure case	Current host heartbeat	What it proves / future acceptance
Healthy host, timer and outbound path; Prometheus stopped	Can continue pinging	Does not detect evaluation loss
Healthy host, timer and outbound path; Alertmanager stopped	Can continue pinging	Does not detect routing/delivery loss
Healthy host; both Prometheus and Alertmanager stopped	Can continue pinging	Does not detect total monitoring-plane loss
Host down, timer stopped, or outbound ping path broken	Pings cease	After Period + Grace, verify Healthchecks DOWN and actual ntfy receipt separately
Healthchecks integration, ntfy provider or phone delivery broken	Pings may still arrive	UP is not evidence that a notification can reach the operator

Future disposable acceptance must distinguish each row, record the last ping and DOWN/UP timestamps plus destination receipt, and restore the test state. Use isolated services/checks and separately approved notification destinations; do not stop production monitoring to validate documentation. A timer miss drill exercises only the cessation/delivery row, not the healthy-host/dead-monitoring rows.

Path	Who delivers to your phone
This dead-man	Healthchecks.io → ntfy (integration in HC UI)
Patch critical / capacity	Homelab curl / Alertmanager → ntfy
Stage 7 pilot notify	OneUptime adapter → separate pilot topics

Ansible only installs the timer and writes /etc/homelab/monitoring-deadman.env with healthchecks_ping_url. It does not post to ntfy.

Prerequisites

• Healthchecks.io account (free plan is enough for one check).
• Phone with the ntfy app (Android / iOS), or any client that can subscribe to https://ntfy.sh/<topic>.
• Age key available where you encrypt SOPS (/etc/homelab/age-key.txt on infra-services, or your operator key).
• Repo checkout with write access so you can commit ciphertext.

1. Pick a dedicated ntfy topic

Use a new topic. Do not reuse:

Existing topic	Purpose
homelab-patch-critical-b4e9	OS patching critical only
homelab-whrrr-capacity-7f2a	Synology volume capacity

Suggested name pattern (match the SOPS example):

homelab-monitoring-deadman-<4hex>

Example: homelab-monitoring-deadman-a1f3.

1. Invent four hex characters (or more) so the topic is hard to guess.
2. Write the full topic name down; you will use it in Healthchecks and on the phone. Optionally record it in ntfy_deadman_topic inside the SOPS file for documentation (Ansible does not send to that topic).

Subscribe on your phone

1. Install the ntfy app from ntfy.sh/app.
2. Subscribe to topic → enter exactly homelab-monitoring-deadman-<4hex> (server left as public ntfy.sh unless you self-host).
3. Optional but recommended for a dead-man:
4. Raise priority / “priority high” for this subscription.
5. Enable override Do Not Disturb / insistent alerting if your phone supports it — this alert means the monitoring plane may be dark.

Prove the topic works (before Healthchecks)

From any machine (Windows PowerShell or bash):

curl -d "homelab monitoring dead-man ntfy smoke" \
  -H "Title: Dead-man ntfy test" \
  -H "Priority: high" \
  -H "Tags: warning,skull" \
  "https://ntfy.sh/homelab-monitoring-deadman-<4hex>"

Expect a push on the phone within a few seconds. If nothing arrives:

• Topic spelling mismatch (most common).
• App subscribed to a different server than ntfy.sh.
• Phone battery / notification permission blocking the app.
• Corporate network blocking ntfy.sh (try mobile data).

Do not continue until this curl test lights up the phone.

2. Create the Healthchecks.io check

1. Sign in at healthchecks.io.
2. Add Check (or Project → Checks → Add).
3. Set timing to match the timer:

Field	Value	Why
Period	5 minutes	Matches OnUnitActiveSec=5min on the timer
Grace	5 minutes	Miss window before DOWN (Period + Grace ≈ 10 minutes)

1. Name it for its measured scope, e.g. infra-services host heartbeat / host timer outbound liveness, not Prometheus plane health.
2. Save, then open the check and copy the Ping URL:

https://hc-ping.com/<uuid>

That UUID is a secret. Treat it like a password — it goes into SOPS, not into chat or screenshots if you can avoid it.

Optional: ping once from your laptop

curl -fsS -o /dev/null -w "%{http_code}\n" "https://hc-ping.com/<uuid>"

Expect 200 and the check flipping to a fresh last-ping in the HC UI. This does not enable the host timer yet; it only proves the URL.

3. Wire Healthchecks to ntfy (the important part)

Healthchecks owns delivery. You are configuring an integration on the check (or project), not Alertmanager and not Ansible.

1. In Healthchecks, open the check (or Project → Integrations).
2. Add / enable the ntfy integration (native integration — not a generic webhook unless you prefer that).
3. Topic / URL fields — use the same topic you subscribed to:

Typical field	Value
Topic or URL	homelab-monitoring-deadman-<4hex> or full https://ntfy.sh/homelab-monitoring-deadman-<4hex>
Priority	high / max if offered
Token	leave empty for a public topic on ntfy.sh

1. Assign the integration to this dead-man check (if the UI is project-wide, restrict or confirm it fires for this check’s DOWN/UP events).
2. Use Healthchecks’ Test / send test notification if available. Otherwise you will prove end-to-end in the Miss drill below.

What you should see on the phone later:

• DOWN (or equivalent) when pings stop past Grace.
• UP / recovered when pings resume.

If curl to ntfy works but HC never notifies:

• Integration not attached to this check.
• Wrong topic in the integration.
• Check still receiving pings (timer still running during a “miss” test).
• HC free-plan / project limits (unlikely for one check).

4. Encrypt the ping URL (SOPS)

File (only this one):

infra/ansible/inventory/group_vars/patch_controller/monitoring-deadman.sops.yaml

Example template (keep in sync when keys change):

infra/ansible/inventory/group_vars/patch_controller/monitoring-deadman.sops.yaml.example

Only healthchecks_ping_url is required for the timer. The ntfy_* keys in the example are documentation / smoke reference — Healthchecks still owns ntfy.

On a machine that can decrypt with the operator or host age key (often infra-services):

cd /path/to/homelab
cd infra/ansible/inventory/group_vars/patch_controller
cp monitoring-deadman.sops.yaml.example monitoring-deadman.sops.yaml

Edit plaintext monitoring-deadman.sops.yaml:

healthchecks_ping_url: "https://hc-ping.com/<uuid>"

ntfy_server: "https://ntfy.sh"
ntfy_deadman_topic: "homelab-monitoring-deadman-<4hex>"

Encrypt in place (do not commit plaintext):

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -e -i monitoring-deadman.sops.yaml

On Windows with WSL or a remote shell, same command once the age key path is correct. Confirm the file is ciphertext (sops: header / encrypted values) before git add.

git add infra/ansible/inventory/group_vars/patch_controller/monitoring-deadman.sops.yaml
git commit -m "Encrypt monitoring dead-man Healthchecks ping URL."
git push

Merge to main (PR or direct per your workflow). Hosts only apply after origin/main updates.

If the file already exists (ciphertext present), edit in place instead of cp:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops monitoring-deadman.sops.yaml
# set healthchecks_ping_url to the new Ping URL, save, exit

5. Apply on infra-services

After merge, wait for ansible-pull or run with the role tag:

# on infra-services, from a checkout of main
cd /opt/homelab
sudo env ANSIBLE_CONFIG=infra/ansible/ansible.cfg \
  ansible-playbook -i infra/ansible/inventory/generated.yml \
  infra/ansible/playbooks/site.yml --limit infra-services \
  --tags monitoring-deadman \
  -e ansible_connection=local

ANSIBLE_CONFIG is what sets roles_path. ansible_connection=local applies the play on this host. Without those, sudo looks for roles beside the playbook and then tries to SSH to root@192.168.6.17.

The role writes /etc/homelab/monitoring-deadman.env (mode 0600) and enables homelab-monitoring-deadman.timer when the ping URL is non-empty. If SOPS is missing or the URL is still REPLACE_…, the timer stays disabled and Ansible logs a warning pointing at this runbook.

6. Verify the happy path

systemctl status homelab-monitoring-deadman.timer
systemctl list-timers homelab-monitoring-deadman.timer
sudo systemctl start homelab-monitoring-deadman.service
journalctl -u homelab-monitoring-deadman.service -n 20 --no-pager

Expect:

• Timer enabled / active.
• Oneshot exit 0 (curl succeeded).
• Healthchecks UI shows a fresh ping within a minute.

Repo smoke helper (on the host):

bash scripts/test-monitoring-deadman.sh
bash scripts/test-monitoring-deadman.sh --live   # starts oneshot when configured

--live only proves the timer env and a ping; it does not exercise ntfy or prove Prometheus/Alertmanager health. Use the separately approved miss drill for cessation and external delivery acceptance.

Miss drill

With owner approval, this tests timer cessation and Healthchecks → ntfy receipt without taking down Prometheus. It does not test Prometheus/Alertmanager loss: they are not prerequisites for the ping. Record the approved window, last ping, DOWN receipt and recovery receipt; restore the timer even if delivery fails.

sudo systemctl stop homelab-monitoring-deadman.timer

1. Wait about Period + Grace (~10 minutes with 5m + 5m).
2. Phone should get a DOWN (or “check is down”) from Healthchecks via ntfy.
3. Healthchecks UI should show the check DOWN.

Recover:

sudo systemctl start homelab-monitoring-deadman.timer
sudo systemctl start homelab-monitoring-deadman.service

1. Healthchecks should flip UP after the next successful ping.
2. Phone should get an UP / recovered notification (if the integration sends recoveries — confirm in HC integration settings if UP is silent).

If DOWN never arrives: re-check §3 and that the timer is actually stopped (systemctl is-active should not be active for the timer).

Troubleshooting

Symptom	Likely cause	What to check
Timer disabled / no env file	SOPS missing or empty URL	Decrypt path; healthchecks_ping_url; role warning in ansible-pull log
Oneshot fails	Bad URL / network	journalctl -u homelab-monitoring-deadman.service; curl the ping URL manually
HC never gets pings	Timer not running or wrong host	Confirm role on infra-services; list-timers
Curl to ntfy works, miss drill silent	HC integration wrong	Topic spelling; integration attached to this check; wait full Grace
Patch/capacity alerts on this topic	Wrong topic reuse	Create a new topic; do not share patch-critical
Phone silent for everything	App / OS	Notification permission; DND; try mobile data

Related

• Synology capacity ntfy — same public ntfy.sh pattern, different topic
• Coordinated OS patching — patch-critical topic homelab-patch-critical-b4e9
• Example secrets: monitoring-deadman.sops.yaml.example
• Role: infra/ansible/roles/monitoring-deadman
• Smoke: scripts/test-monitoring-deadman.sh