Runbooks

Operational procedures for common tasks.

• Alert triage — Alertmanager, Grafana, Prometheus, and runbook drilldowns
• Prox memory soak — 24h attributed Prox RAM samples before shrink/retire
• Operations platform migration — staged OneUptime rollout, gates, and rollback
• Monitoring dead-man — Healthchecks.io external heartbeat + ntfy
• Backup alert triage — restic metrics, timers, copy targets, and restore-test alerts
• Coordinated OS patching — Phase 8 weekly apt upgrades from infra-services
• Saltbox container updates — images and sb install on saltierpoop (not Komodo)
• Secrets — bootstrap, rotation, compromise response
• Adding a Service — onboarding a new Compose stack
• Restore — per-service restic restore procedures
• OneUptime backup and restore — tier-1 logical backups and isolated drill
• OneUptime upgrade and rollback — migration-safe release procedure
• DR from Zero — full lab rebuild from offsite backup + this repo
• Public edge incidents (SEC-009) — Cloudflare token leak, Traefik ACME, Authentik DB
• Synology capacity ntfy — Whrrr volume alerts via ntfy.sh
• Cloudflare Pages (docs deploy) — mkdocs → hldocs-c0acdec9.pages.dev
• Central GitHub runner management — dispatcher how-to matrix, revive offline runners, watchdog, qualify, guarded maintenance
• Historical GitHub runner fleet — retired WSL-era reference; not an operating procedure
• GitHub Actions runner observability — runner fleet, jobs, host metrics, logs, and alerts
• Decommission Old Monitoring — LXC 101/108/112 migration and teardown
• Phase 7 Owner Actions — firewall, DSM, WiFi, Tailscale, AdGuard, PiHole decom
• Roborock on IoT (VLAN 5) — DNS/ZBF/AdGuard when cloud vacuums go offline
• Network Observations (2026-06-03) — live-scan anomalies & connectivity triage
• Phase 7R Audit Questionnaire — takeover audit owner Q&A (2026-06-18)
• Phase 7R ZBF Remediation — UDM policies from audit decisions
• Documentation validation — verify docs match live state
• JDownloader2 over Mullvad (Saltbox) — Gluetun + sandbox-jdownloader2 on saltierpoop
• qBittorrent over Mullvad (Saltbox) — Gluetun-shared qbittorrent (replace qbittorrentvpn)
• SSH keys and infra-services — per-tool keys, Cursor vs deploy, GitHub from host
• Inbound SSH (humans vs agents) — 1Password vs IDE keys on infra-services
• Saltierpoop inbound SSH (agents) — saltierpoop-cursor + patch-controller jump
• Whrrr VMM inbound SSH (agents) — ubuncap-cursor / recordurbate-cursor
• Authentik cross-host SSO — ADR-002 implementation; outposts on infra-services
• Authentik infra admin setup — owner walkthrough: provider, outpost, token, verify
• Komodo Authentik OIDC — single-login for Komodo (native OIDC, not forward-auth)
• Komodo GitHub webhook relay — push-to-deploy via self-hosted Actions
• Komodo connect LXC server — periphery + /opt/homelab for phoenix/sonarqube
• Grafana Authentik auth proxy — single-login for Grafana (forward-auth + auth proxy)
• Infra single login (index) — status for all infra apps
• AdGuard edge SSO — disable UI auth behind forward-auth
• Wazuh edge SSO — OpenSearch proxy auth when stack is deployed
• Proxmox API Token — create labctl token with guest-agent exec for VM interrogation
• Guest discovery — Proxmox/VMM reconciliation; operator SSH via sync-discovery-lxc-operator-keys.py
• Compute decommission queue — backup + destroy order after disposition review
• Central syslog (Graylog) — Pattern E rollout complete (2026-06-26)
• HAOS retention and Ford timeline — 15mo recorder + Mach-E map dashboard
• Wazuh SIEM — security monitoring on managed Linux
• infra-services capacity & resize — RAM/disk targets, Proxmox VM 123 resize
• Metrimon decommission gate — pre-destroy checklist for VM 106
• LLM observability — LiteLLM gateway + Phoenix traces
• Telegram Companion owner E2E — lab slice for Mac archive app (full checklist in Companion)