Runbooks

Operational procedures for common tasks.

• Coordinated OS patching — Phase 8 weekly apt upgrades from infra-services
• Secrets — bootstrap, rotation, compromise response
• Adding a Service — onboarding a new Compose stack
• Restore — per-service restic restore procedures
• DR from Zero — full lab rebuild from offsite backup + this repo
• Public edge incidents (SEC-009) — Cloudflare token leak, Traefik ACME, Authentik DB
• Synology capacity ntfy — Whrrr volume alerts via ntfy.sh
• Cloudflare Pages (docs deploy) — mkdocs → hldocs-c0acdec9.pages.dev
• GitHub self-hosted runners — homelab-ci (WSL) + homelab-lan (infra-services)
• Decommission Old Monitoring — LXC 101/108/112 migration and teardown
• Phase 7 Owner Actions — firewall, DSM, WiFi, Tailscale, AdGuard, PiHole decom
• Roborock on IoT (VLAN 5) — DNS/ZBF/AdGuard when cloud vacuums go offline
• Network Observations (2026-06-03) — live-scan anomalies & connectivity triage
• Phase 7R Audit Questionnaire — takeover audit owner Q&A (2026-06-18)
• Phase 7R ZBF Remediation — UDM policies from audit decisions
• Documentation validation — verify docs match live state
• JDownloader2 over Mullvad (Saltbox) — Gluetun + sandbox-jdownloader2 on saltierpoop
• qBittorrent over Mullvad (Saltbox) — Gluetun-shared qbittorrent (replace qbittorrentvpn)
• SSH keys and infra-services — per-tool keys, Cursor vs deploy, GitHub from host
• Inbound SSH (humans vs agents) — 1Password vs IDE keys on infra-services
• Saltierpoop inbound SSH (agents) — saltierpoop-cursor + patch-controller jump
• Authentik cross-host SSO — ADR-002 implementation; outposts on infra-services
• Authentik infra admin setup — owner walkthrough: provider, outpost, token, verify
• Komodo Authentik OIDC — single-login for Komodo (native OIDC, not forward-auth)
• Komodo GitHub webhook relay — push-to-deploy via self-hosted Actions
• Grafana Authentik auth proxy — single-login for Grafana (forward-auth + auth proxy)
• Infra single login (index) — status for all infra apps
• AdGuard edge SSO — disable UI auth behind forward-auth
• Wazuh edge SSO — OpenSearch proxy auth when stack is deployed
• Proxmox API Token — create labctl token with guest-agent exec for VM interrogation
• Guest discovery — Proxmox/VMM reconciliation, power windows, in-guest facts
• Compute decommission queue — backup + destroy order after disposition review
• Central syslog (Graylog) — Pattern E rollout complete (2026-06-26)
• Wazuh SIEM — security monitoring on managed Linux
• infra-services capacity & resize — RAM/disk targets, Proxmox VM 123 resize
• Metrimon decommission gate — pre-destroy checklist for VM 106