Coordinated OS Patching (Phase 8 pass-1)

Operational guide for push-mode Linux patching from infra-services, including observability and notifications. Architecture detail: patching architecture.

---

Status at a glance

Area	In repo (pass-1)	Live on homelab
C&C timer + SSH key	Yes	Yes (infra-services)
Wave patching (saltierpoop → prox → infra-services)	Yes	Yes (live run 2026-06-23)
Discord weekly summary + rich embeds	Yes	Yes
ntfy critical (failure, pre-reboot)	Yes	Yes
Prometheus patch alerts	Yes	After monitoring stack reload
Grafana patching dashboard	Yes	Yes (/d/homelab-patching)
ansible_pull.prom fix	Yes	Yes (all pull hosts)
ARA playbook recording	Yes	Yes (ara.infra.realemail.app)
Weekly schedule (Sun 04:00 PT)	Yes	Timer enabled

Pass-1 is live. The Owner checklist below is kept as deploy reference; items were completed 2026-06-23.

---

Assigned ntfy topic

Use this topic for patch-critical alerts only (separate from Whrrr capacity homelab-whrrr-capacity-7f2a):

homelab-patch-critical-b4e9

Server: https://ntfy.sh (public ntfy.sh — same as capacity alerts).

Subscribe on your phone now (takes 30 seconds):

1. Install ntfy app.
2. Add subscription → topic homelab-patch-critical-b4e9.
3. Test (basic):

curl -d "homelab patch ntfy test" -H "Title: Patch test" \
  "https://ntfy.sh/homelab-patch-critical-b4e9"

1. Test enriched notifications (orchestrator, pre-reboot, Alertmanager template):

# From repo root on infra-services (notify env is mode 600 root-only):
sudo bash -c 'set -a; source /etc/homelab/patching-notify.env; set +a; bash /var/lib/ansible-pull/homelab/scripts/test-patch-ntfy.sh --live'

CI runs scripts/test-patch-ntfy.sh --dry-run on every push. Live mode sends three TEST messages; expect skull/patch tags, ARA/Grafana action buttons on failure-style, and a formatted Alertmanager alert.

Enriched headers used in production:

Path	Tags	Click	Actions
Orchestrator failure	skull,patch	ARA	Open ARA, Grafana
Pre-reboot	rotating_light,computer	Grafana patching dashboard	—
Alertmanager critical patching	(template)	(template)	—

Alertmanager posts to ?template=alertmanager so webhook JSON becomes a readable title + message instead of raw JSON on your phone.

Discord uses rich embeds (color, fields, timestamps) and link buttons via /usr/local/lib/homelab/discord-patch-notify.py:

sudo bash -c 'set -a; source /etc/homelab/patching-notify.env; set +a; bash /var/lib/ansible-pull/homelab/scripts/test-patch-discord.sh --live'

Expect four TEST embeds: converge heartbeat (blue), success (green), failure (red), reboot (yellow) — each with ARA / Grafana / Runbook buttons.

---

What pass-1 built (reference)

Patching mechanism

Component	Path / unit
Patch controller role	infra/ansible/roles/patch-controller/
Patching role	infra/ansible/roles/patching/
Playbook	infra/ansible/playbooks/patch.yml
Systemd timer	homelab-patch-orchestrate.timer
Systemd service	homelab-patch-orchestrate.service
Wrapper script	/usr/local/bin/homelab-patch-orchestrate
SSH key	/etc/homelab/patch-controller/id_ed25519
Notify env	/etc/homelab/patching-notify.env

Wave order: saltierpoop (0) → Whrrr VMM (ubuncap, recordurbate) → Proxmox LXCs → prox hypervisor (1) → infra-services (2). C&C patches itself last.

Policy defaults (group_vars/patching_targets.yml):

• patching_upgrade_type: security — apt safe upgrade
• patching_reboot: auto_if_required — reboot if /var/run/reboot-required
• unattended-upgrades removed on targets

Observability

Signal	Source
patch.prom	Each patching target — last success time, reboot flag
homelab-sync-patch-reboot-metric	Boot oneshot + ansible-pull ExecStartPost — keeps reboot gauge matched to /var/run/reboot-required
patch_orchestrate.prom	infra-services only — orchestrator exit code + run time
ansible_pull.prom	Each ansible-pull host — last successful apply
ARA	All orchestrator / playbook runs
Grafana	Dashboard Coordinated OS Patching (patching.json)

Notifications

Event	Discord	ntfy
Weekly patch success	Rich embed + per-host fields + link buttons	—
Orchestrator / playbook failure	Rich embed + @here + ARA/Grafana/Prometheus buttons	Rich (tags, click, actions)
Pre-reboot on a host	Rich embed + Grafana button	Rich (tags, click)
Prometheus patch alerts	Enhanced markdown + runbook links	Alertmanager template (critical only)
Converge heartbeat	Rich embed once per 24h on ansible-pull (git SHA, pull time, ntfy status)	Enriched curl probe

Stamp file: /var/lib/homelab/.patch-discord-validation-sent (24h default). Force an immediate post with host var patching_notify_discord_validation_force: true.

Alerts (monitoring/prometheus/alerts/patching.yml):

Alert	Severity	Condition
OSPatchStale	warning	last successful security patch is over 8 days old or missing
OSPatchOrchestrateFailed	critical	orchestrator exit code ≠ 0
PatchRebootPending	warning	reboot-required flag set 30m

---

Owner checklist

Complete in order. Checked off 2026-06-23 (pass-1 live).

• [x] Subscribe to ntfy topic homelab-patch-critical-b4e9 (see above)
• [x] Push pass-1 to main if not already on origin
• [x] SOPS — Ansible notify (Discord webhook + ntfy topic):

cd infra/ansible/inventory/group_vars/patch_controller
cp notify.sops.yaml.example notify.sops.yaml
# Edit: patching_notify_discord_webhook_url = same URL as monitoring Discord webhook
#       patching_notify_ntfy_topic = homelab-patch-critical-b4e9  (already in example)
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -e -i notify.sops.yaml
git add notify.sops.yaml && git commit && git push

• [x] SOPS — Alertmanager (add to existing monitoring secrets):

NTFY_SERVER: "https://ntfy.sh"
NTFY_PATCH_TOPIC: "homelab-patch-critical-b4e9"

Re-decrypt .env on infra-services and reload alertmanager (below). - [x] Converge all managed hosts — wait for ansible-pull or trigger on each: - infra-services, saltierpoop, prox (ansible-pull + ARA client) - [x] Reload monitoring on infra-services - [x] Publish patch-controller.pub and verify SSH from infra-services (sudo — key is root-only):

sudo ssh -i /etc/homelab/patch-controller/id_ed25519 -o IdentitiesOnly=yes someone@192.168.6.243 true
sudo ssh -i /etc/homelab/patch-controller/id_ed25519 -o IdentitiesOnly=yes root@192.168.6.71 true

• [x] Dry-run patch (see Testing)
• [x] One live patch via systemd unit (not raw playbook)
• [x] Notification test matrix (see Testing)
• [x] Mark README Owner TODO rows for Phase 8 done

---

Prerequisites

1. ansible-pull converging on infra-services (deploys patch-controller role)
2. Patch-controller SSH from infra-services to saltierpoop (someone) and prox (root) — verified with the patch private key (see below). infra-services has no general outbound SSH to other hosts; one-time bootstrap uses an operator workstation (scripts/bootstrap-patch-ssh-from-operator.sh).
3. Repo checkout at /var/lib/ansible-pull/homelab (or /opt/homelab)
4. SOPS age key on infra-services at /etc/homelab/age-key.txt

After bootstrap, targets also trust the committed infra/ansible/inventory/files/patch-controller.pub via each host's ansible-pull (from="192.168.6.17"). Optional backup-fetch push to saltierpoop only runs when /etc/homelab/backup-fetch/id_ed25519 parses cleanly (ssh-keygen -y).

---

Deploy steps (infra-services)

1. Verify patch-controller landed

ssh someone@192.168.6.17
sudo systemctl list-timers homelab-patch-orchestrate.timer
sudo ls -la /etc/homelab/patch-controller/
sudo ls -la /etc/homelab/patching-notify.env   # after notify.sops.yaml committed + pull

1b. Patch SSH bootstrap (one-time, if not already done)

From a workstation with SSH to all hosts (not from infra-services as someone):

./scripts/bootstrap-patch-ssh-from-operator.sh

Or verify manually on infra-services:

sudo ssh -i /etc/homelab/patch-controller/id_ed25519 -o IdentitiesOnly=yes someone@192.168.6.243 true
sudo ssh -i /etc/homelab/patch-controller/id_ed25519 -o IdentitiesOnly=yes root@192.168.6.71 true

patch-controller.pub in git keeps new targets converged via ansible-pull.

Expected:

• Timer: Sun … weekly, timezone America/Los_Angeles
• Private key: /etc/homelab/patch-controller/id_ed25519 (mode 600)
• Notify env: mode 600, contains NTFY_PATCH_TOPIC=homelab-patch-critical-b4e9

2. Reload monitoring (after SOPS + push)

cd /opt/homelab/services/monitoring
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d .env.sops.yaml \
  | sed 's/: /=/' > .env
docker compose up -d alertmanager prometheus grafana

3. Confirm metrics paths (after converge)

On each managed host:

ls -la /var/lib/node_exporter/textfile_collector/patch.prom
ls -la /var/lib/node_exporter/textfile_collector/ansible_pull.prom

On infra-services only:

ls -la /var/lib/node_exporter/textfile_collector/patch_orchestrate.prom

Prometheus: https://prometheus.infra.realemail.app — query homelab_patch_last_success_unixtime.

---

Testing

Bootstrap patch SSH (one-time)

Diagnose backup-fetch first (error in libcrypto = unreadable key file, not “wrong host key”):

sudo ls -la /etc/homelab/backup-fetch/id_ed25519
sudo ssh-keygen -y -f /etc/homelab/backup-fetch/id_ed25519
# Must match infra/ansible/inventory/files/backup-fetch.pub

If ssh-keygen -y fails, fix or regenerate backup-fetch before relying on automated saltierpoop push. After ansible-pull redeploys the key, re-run ssh-keygen -y. See scripts/update-backup-sops.sh if SOPS and disk diverged.

saltierpoop (recommended — uses your normal someone SSH, not backup-fetch):

PUB=$(sudo cat /etc/homelab/patch-controller/id_ed25519.pub)
ssh someone@192.168.6.243 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && (grep -qxF '$PUB' ~/.ssh/authorized_keys 2>/dev/null || echo '$PUB' >> ~/.ssh/authorized_keys) && chmod 600 ~/.ssh/authorized_keys"

prox — one-time password auth (patch private key is root-only):

sudo ssh-copy-id -i /etc/homelab/patch-controller/id_ed25519 -o IdentitiesOnly=yes root@192.168.6.71

Verify patch key (sudo required):

sudo ssh -i /etc/homelab/patch-controller/id_ed25519 -o IdentitiesOnly=yes someone@192.168.6.243 true
sudo ssh -i /etc/homelab/patch-controller/id_ed25519 -o IdentitiesOnly=yes root@192.168.6.71 true

Dry run (check mode)

Does not send Discord summary, reboot, or update all metrics reliably. Use to preview apt changes only.

cd /var/lib/ansible-pull/homelab/infra/ansible
sudo bash -c '
  export ANSIBLE_PRIVATE_KEY_FILE=/etc/homelab/patch-controller/id_ed25519
  export ANSIBLE_CONFIG=$PWD/ansible.cfg
  ansible-playbook -i inventory/generated.yml playbooks/patch.yml --check --diff
'

Pick a low-traffic window for the first live run (saltierpoop is wave 0).

Live patch (production path)

Always use the systemd unit so the wrapper records exit metrics and failure notifications:

sudo systemctl start homelab-patch-orchestrate.service
sudo journalctl -u homelab-patch-orchestrate.service -n 100 --no-pager

Runs appear in ARA: https://ara.infra.realemail.app. Machine callbacks use the allowlisted https://ara.infra.realemail.app/api/ route; ARA no longer publishes a direct host :8000 port.

If ARA is empty, see services/ara/README.md (ara pip package, ANSIBLE_CALLBACK_PLUGINS, ara_default callback).

bash /var/lib/ansible-pull/homelab/scripts/test-ara-client.sh --live   # infra-services

Notification test matrix (pass-1 acceptance)

#	Test	How	Expected
0	ntfy enriched (automated)	bash scripts/test-patch-ntfy.sh --dry-run (CI) or --live on infra-services	Dry-run passes; live sends 3 TEST pushes with tags/click/actions
0b	Discord rich (automated)	bash scripts/test-patch-discord.sh --dry-run (CI) or --live on infra-services	Four TEST embeds with colors + link buttons
1	ntfy subscribe	curl test above or --live script	Phone notification
2	Successful live patch	systemctl start homelab-patch-orchestrate.service	Discord summary with lines like saltierpoop: patched; no ntfy
3	Metrics fresh	Prometheus / Grafana	patch.prom, patch_orchestrate.prom updated; exit code 0
4	No false stale	Prometheus alerts	OSPatchStale not firing
5	MAINTENANCE skip	sudo touch /etc/homelab/MAINTENANCE on saltierpoop; run patch; remove file	Summary shows saltierpoop: skipped_maintenance; others patch
6	Orchestrator failure	Temporarily break SSH to prox or edit wrapper to use bad inventory; run unit; revert	Discord + ntfy critical (skull/patch tags, ARA click); homelab_patch_orchestrate_last_exit_code != 0
7	Pre-reboot ntfy	Only if a kernel update leaves /var/run/reboot-required	ntfy before reboot (rotating_light tag, Grafana click)

Discord summary example

Success posts a green embed with inline fields per host (not plain text):

✅ Weekly OS Patch Complete
Coordinated patching finished across all waves. · 2 patched · 1 rebooted

saltierpoop          prox                 infra-services
✅ Patched            🔄 Rebooted          ✅ Patched

Buttons: 📋 ARA · 📊 Grafana · 📖 Runbook

With MAINTENANCE on one host:

saltierpoop: skipped_maintenance
prox: patched
infra-services: patched

---

Operations

Pause patching on one host

sudo touch /etc/homelab/MAINTENANCE   # on target
sudo rm /etc/homelab/MAINTENANCE     # when done

Also pauses ansible-pull on that host.

Pause all patching (C&C)

On infra-services only:

sudo touch /etc/homelab/MAINTENANCE

Orchestrator will not start (ExecCondition). Manual playbook still possible.

Change schedule

Edit patch_orchestrate_calendar / patch_orchestrate_timezone in roles/patch-controller/defaults/main.yml, commit, wait for ansible-pull.

Default: Sunday 04:00 Pacific, RandomizedDelaySec=15min.

Change upgrade policy

Goal	Variable	Value
Security updates only (default)	patching_upgrade_type	security
Full dist-upgrade	patching_upgrade_type	dist
Never auto-reboot	patching_reboot	never
Reboot if required	patching_reboot	auto_if_required

Set in group_vars/patching_targets.yml or per-host host_vars/.

Onboard a new Linux host

1. ansible.managed: true in inventory; add to patching_targets + patching_waveN
2. uv run python inventory/generators/render-ansible.py and commit
3. Bootstrap with site.yml (ansible-pull)
4. Converge infra-services (patch-controller key authorize play)
5. Add wave play in playbooks/patch.yml if you created a new wave group

Redistribute patch-controller SSH key

If infra-services was rebuilt:

cd /var/lib/ansible-pull/homelab/infra/ansible
export ANSIBLE_CONFIG=$PWD/ansible.cfg
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt \
sudo -E ansible-playbook -i inventory/generated.yml playbooks/site.yml \
  --tags patch-controller \
  --limit infra-services \
  -e ansible_connection=local

Whrrr VMM guests (wave 0 whrrr)

ubuncap and recordurbate are Ansible-managed VMM guests on whrrr (not customer-app hosts). OS patching uses someone + NOPASSWD sudo.

One-time from operator workstation (SSH to VMs already works):

./scripts/bootstrap-patch-ssh-from-operator.sh

Then on infra-services:

cd /opt/homelab
sudo env ANSIBLE_CONFIG=infra/ansible/ansible.cfg \
  ANSIBLE_PRIVATE_KEY_FILE=/etc/homelab/patch-controller/id_ed25519 \
  ansible-playbook -i infra/ansible/inventory/generated.yml \
  infra/ansible/playbooks/bootstrap-whrrr-vmm-secrets.yml

sudo env ANSIBLE_CONFIG=infra/ansible/ansible.cfg \
  ANSIBLE_PRIVATE_KEY_FILE=/etc/homelab/patch-controller/id_ed25519 \
  ansible-playbook -i infra/ansible/inventory/generated.yml \
  infra/ansible/playbooks/site.yml --limit ubuncap,recordurbate

Customer-app containers (recordurbate-tiktok) stay on ubuncap — homelab does not patch those images.

Cursor/agent SSH: Whrrr VMM inbound SSH agents.

Proxmox LXC guests (wave 1 guests)

Active LXCs with their own OS (graylog, pulse, octoprint, harbor-registry) are ansible.managed: true in inventory, in patching_wave1_guests (patched before the prox hypervisor). Group defaults: group_vars/proxmox_guests.yml (root SSH).

One-time from infra-services after inventory is committed:

cd /opt/homelab
sudo env ANSIBLE_CONFIG=infra/ansible/ansible.cfg \
  ANSIBLE_PRIVATE_KEY_FILE=/etc/homelab/patch-controller/id_ed25519 \
  ansible-playbook -i infra/ansible/inventory/generated.yml \
  infra/ansible/playbooks/bootstrap-discovery-ssh.yml -l prox \
  --tags bootstrap-pct

sudo env ANSIBLE_CONFIG=infra/ansible/ansible.cfg \
  ANSIBLE_PRIVATE_KEY_FILE=/etc/homelab/patch-controller/id_ed25519 \
  ansible-playbook -i infra/ansible/inventory/generated.yml \
  infra/ansible/playbooks/site.yml \
  --limit graylog,pulse,octoprint,harbor-registry

bootstrap-discovery-ssh installs the patch-controller pubkey via pct and copies /root/.ssh/github_deploy and /etc/homelab/age-key.txt from prox into each LXC (needed for ansible-pull, same pattern as prox hypervisor bootstrap).

prox bootstrap

prox is root-only (SSH alias proxbox). ansible-pull is live as of 2026-06-23. host_vars/prox.yml sets common_ssh_permit_root: prohibit-password and skips systemd-timesyncd (PVE has no timesyncd unit).

First-time bootstrap from infra-services (patch-controller key):

cd /var/lib/ansible-pull/homelab
sudo env ANSIBLE_CONFIG=infra/ansible/ansible.cfg \
  ANSIBLE_PRIVATE_KEY_FILE=/etc/homelab/patch-controller/id_ed25519 \
  ansible-playbook -i infra/ansible/inventory/generated.yml \
  infra/ansible/playbooks/site.yml --limit prox --tags ansible-pull

Also required on prox (not in git): /etc/homelab/age-key.txt, GitHub deploy key under /root/.ssh/ (copy pattern from infra-services bootstrap).

---

Troubleshooting

Symptom	Likely cause	Fix
Root SSH to prox suddenly fails	ansible-pull set PermitRootLogin no	Proxmox console or UI shell; set prohibit-password in sshd; pull host_vars/prox.yml
Permission denied (publickey) to prox	patch-controller pubkey not on root	Commit patch-controller.pub + ansible-pull on prox, or sudo ssh-copy-id -i /etc/homelab/patch-controller/id_ed25519 root@192.168.6.71
ansible-pull on prox: SOPS / tailscale fail	Missing /etc/homelab/age-key.txt	Copy automation age key from infra-services; see infra/ansible/pull/bootstrap.sh
ARA empty / callback skipped	Missing ara pip or ara-callback.env	ansible-pull ara-client.yml; scripts/test-ara-client.sh --live
Load key ... error in libcrypto (backup-fetch)	Corrupt private key on disk or bad SOPS deploy	sudo ssh-keygen -y -f /etc/homelab/backup-fetch/id_ed25519; fix SOPS + ansible-pull, or bootstrap patch SSH via normal someone SSH
Identity file ... not accessible: Permission denied	Manual test as someone	Patch key is root-only; prefix commands with sudo
Patch playbook UNREACHABLE on infra-services	C&C play used SSH with patch key	Pull latest main; C&C plays use ansible_connection: local
Discord pre-reboot UNREACHABLE root@ C&C IP	delegate_to: patch_controller without local connection (inherits remote ansible_user)	notify-discord-reboot.yml must set ansible_connection: local on those tasks — never SSH hairpin
Patch skipped on host	MAINTENANCE flag	test -f /etc/homelab/MAINTENANCE
No Discord summary	Missing / invalid notify.sops.yaml or webhook	Check /etc/homelab/patching-notify.env; verify SOPS committed
No ntfy on failure	Topic not in env or not subscribed	grep NTFY_PATCH_TOPIC in notify env + monitoring .env; test curl
OSPatchStale firing	Patch hasn't run in 8+ days, or patch.prom wrote a stale success epoch	Check timer / last orchestrate; on target date +%s vs homelab_patch_last_success_unixtime — if mtime is fresh but epoch is old, rewrite gauge (do not delete the file) and ensure patching metrics use live date +%s not cached ansible_date_time
OSPatchOrchestrateFailed	Last wrapper exit ≠ 0	journalctl -u homelab-patch-orchestrate; fix playbook error; re-run
PatchRebootPending	OS flag still set, or stale patch.prom after reboot	See PatchRebootPending triage
AnsiblePullStale	ansible-pull not applying	systemctl status ansible-pull-apply.timer; check git pull / SSH URL
No patch.prom in Prometheus	node_exporter textfile not enabled	ansible-pull converge common role on host
saltierpoop media blip	Wave 0 runs first	Schedule is off-peak Sun 04:00; manual runs: pick quiet time
prox reboot concern	Hypervisor restart	Safe upgrade rarely requires reboot; ntfy warns if it does
Raw playbook vs unit	Ran ansible-playbook directly	Use systemctl start homelab-patch-orchestrate.service for metrics + failure notify

PatchRebootPending triage

Discord keeps firing because Prometheus alert PatchRebootPending watches the textfile metric homelab_patch_reboot_required == 1 (for 30m), not the live /var/run/reboot-required file.

Automatic sync (preferred): homelab-sync-patch-reboot-metric rewrites only the reboot gauge from the live OS flag. It runs:

• on every boot (homelab-sync-patch-reboot-metric.service oneshot)
• after each successful ansible-pull-apply (ExecStartPost)

That clears the common infra-services case (deferred shutdown -r +2 writes patch.prom with gauge 1 before reboot). Manual clear below is the fallback if sync is missing or failed.

flowchart TD
  A[PatchRebootPending Discord] --> B{SSH host: does /var/run/reboot-required exist?}
  B -->|yes| C[Real pending reboot]
  B -->|no| D{patch.prom still says reboot_required 1?}
  C --> E[sudo reboot or finish hung reboot]
  D -->|yes| F[Run sync script or clear gauge]
  D -->|no| G[Wait scrape / check Prometheus instance label]
  E --> H[Confirm flag gone; sync if still 1]
  F --> I[Alert resolves after scrape]
  H --> I

1. Confirm which host and what Discord is reacting to

Check	Where
Alert instance	Discord embed / Alertmanager / Grafana patching
Live OS flag	SSH target: ls /var/run/reboot-required
Metric on disk	SSH target: cat /var/lib/node_exporter/textfile_collector/patch.prom
Metric in Prom	homelab_patch_reboot_required{instance="<host>"}

Entry point from other alerts: Alert triage → Patching.

2. Decide: real reboot still needed vs stale metric

On the alerted host:

# OS still wants a reboot?
ls -la /var/run/reboot-required /var/run/reboot-required.pkgs 2>&1

# What Prometheus is scraping (may disagree with OS after reboot)
cat /var/lib/node_exporter/textfile_collector/patch.prom

# Already on the expected kernel? (1 = OK on Ubuntu needrestart -b)
uptime; who -b
command -v needrestart >/dev/null && sudo needrestart -b | egrep 'NEEDRESTART-K(CUR|EXP|STA)'

Host state	Meaning	Action
/var/run/reboot-required exists	Real pending reboot	Reboot (below)
Flag missing, homelab_patch_reboot_required 1 in patch.prom	Stale metric (common on infra-services before sync)	Run sync / clear
Flag missing, metric already 0, alert still firing	Scrape lag / wrong instance	Wait ~1–2m; confirm Prom query matches host

Why infra-services can go stale without sync: wave 2 uses ansible_connection: local. The role schedules shutdown -r +2, then writes patch.prom with homelab_patch_reboot_required 1 before the reboot. After reboot, /var/run/reboot-required is gone (tmpfs). Remote hosts refresh the gauge after ansible.builtin.reboot returns; C&C cannot — the boot / ansible-pull sync closes that gap.

3. Clear a stale reboot gauge

Only when /var/run/reboot-required is absent and the host is healthy (uptime recent, kernel matches). Prefer the sync script (preserves last-success):

# Preferred — same path boot / ansible-pull use
sudo /usr/local/bin/homelab-sync-patch-reboot-metric
cat /var/lib/node_exporter/textfile_collector/patch.prom

# If the script is missing (pre-converge), force the gauge:
PROM=/var/lib/node_exporter/textfile_collector/patch.prom
sudo sed -i 's/^homelab_patch_reboot_required .*/homelab_patch_reboot_required 0/' "$PROM"
cat "$PROM"

Confirm the helper is installed and enabled:

systemctl status homelab-sync-patch-reboot-metric.service --no-pager
ls -la /usr/local/bin/homelab-sync-patch-reboot-metric

Do not delete patch.prom to silence the alert — the missing success metric trips OSPatchStale directly.

4. When the OS flag is still present

# See which packages requested the reboot
cat /var/run/reboot-required.pkgs 2>/dev/null

# Prefer a clean reboot over force
sudo reboot

If reboot already ran but the flag returned immediately, check whether apt or another tool recreated it (pending kernel still not active, failed boot into new kernel, or a package still marking reboot-required). Use needrestart -b and uname -r vs the expected kernel.

After a successful reboot, re-check step 2 — if the OS flag is gone but patch.prom still says 1, clear the gauge (step 3).

5. Verify Discord stops

homelab_patch_reboot_required{instance="infra-services"}  # or the alerted instance

Expect 0. In Alertmanager, PatchRebootPending should leave firing within one scrape + evaluation cycle (~1–2 minutes after the 30m for is no longer satisfied). No silence needed once the gauge is correct.

Useful commands

# Orchestrator logs
sudo journalctl -u homelab-patch-orchestrate.service -n 200 --no-pager

# Timer next run
systemctl list-timers homelab-patch-orchestrate.timer

# Last orchestrator metrics
cat /var/lib/node_exporter/textfile_collector/patch_orchestrate.prom

# Test Discord (four rich TEST embeds; notify env is root-only)
sudo bash -c 'set -a; source /etc/homelab/patching-notify.env; set +a; bash /var/lib/ansible-pull/homelab/scripts/test-patch-discord.sh --live'

# Test ntfy (basic)
curl -d "test" -H "Title: Patch" \
  "https://ntfy.sh/homelab-patch-critical-b4e9"

# Test enriched ntfy (three TEST notifications; notify env is root-only)
sudo bash -c 'set -a; source /etc/homelab/patching-notify.env; set +a; bash /var/lib/ansible-pull/homelab/scripts/test-patch-ntfy.sh --live'

---

Future consideration: Discord forum threads

Pass-1 posts every patch notification as a top-level message in the alerts channel. That is intentional (simple webhook, no state). Discord forum channels offer a richer model that stays webhook-only — no bot required.

Idea

One forum post per orchestrator run; wave updates, reboot warnings, success, and failure all land inside that thread instead of cluttering the channel timeline.

#homelab-patching (forum channel)
└── Patch 2026-06-23 04:09          ← thread_name on kickoff
    ├── Wave 0: saltierpoop done
    ├── Wave 1: prox done
    ├── Reboot imminent: prox
    └── ✅ Weekly OS Patch Complete

Webhook mechanics (Discord API)

Action	How
Create thread	POST webhook with thread_name (forum or media channel only)
Post into thread	?thread_id=<snowflake> on execute URL
Get thread id back	?wait=true on first POST; response includes thread_id

Regular text channels cannot create threads via webhook alone — that needs a bot with CREATE_PUBLIC_THREADS. Forum channel is the low-friction path.

Implementation sketch (not built)

1. Owner: convert alerts channel to Forum (or dedicated #patch-runs forum).
2. Orchestrator wrapper: on start, POST kickoff with thread_name + ?wait=true; write thread_id to e.g. /run/homelab-patch-discord-thread.
3. Extend discord_patch_notify.py to accept --thread-id (append query param).
4. Pre-reboot, summary, failure: all use stored thread_id; clear file on exit.
5. Alertmanager stays out-of-thread unless a custom webhook receiver reads the active thread_id (native discord_configs has no thread support).
6. Converge heartbeat: once per 24h top-level post (or a test forum); set patching_notify_discord_validation_force: true to re-probe immediately.

Trade-offs

Pro	Con
Clean channel; one collapsible thread per week	Needs forum channel + wait=true state
Matches “one patch run = one story”	Alertmanager alerts don’t auto-thread
Webhook-only (no new service)	Slightly more wrapper complexity

Status: documented for future cohort; pass-1 keeps flat timeline by design.

---

Out of scope (pass-1)

• Saltbox container image updates (Saltbox / Komodo)
• Loki/journald shipping for orchestrator logs (Phase 9 syslog cohort)
• Per-wave Discord messages (summary-only design; see Future consideration: Discord forum threads for threaded alternative)
• Appliances (UDM, DSM, HAOS) — vendor patching

---

Related

• Patching architecture
• Synology capacity ntfy — same ntfy.sh pattern, different topic
• PLAN.md Phase 8
• secrets/patching/README.md
• Grafana: Coordinated OS Patching dashboard