Runbooks

Choose a procedure by the task at hand. For topology, design, and dated infrastructure evidence, see Infrastructure.

Incident response

• Alert triage — Alertmanager, Grafana, Prometheus, and runbook drilldowns
• Public edge incidents (SEC-009) — Cloudflare token leak, Traefik ACME, Authentik DB
• Komodo periphery stale Docker socket — recover deploys after a Docker restart

Backup and recovery

• Backup alert triage — restic metrics, timers, copy targets, and restore-test alerts
• Restore — per-service restic restore procedures
• OneUptime backup and restore — tier-1 logical backups and isolated drill
• DR from Zero — full lab rebuild from offsite backup and this repo

Access and identity

Secrets, SSH, and API access

• Secrets — bootstrap, rotation, compromise response
• SSH keys and infra-services — per-tool keys, Cursor vs deploy, GitHub from host
• Inbound SSH (humans vs agents) — 1Password vs IDE keys
• Saltierpoop inbound SSH (agents) — saltierpoop-cursor and patch-controller jump
• Whrrr VMM inbound SSH (agents) — ubuncap-cursor / recordurbate-cursor
• Proxmox API token — create a labctl token with guest-agent exec for VM interrogation

Single sign-on

• Infra single login (index) — status for all infra apps
• Authentik cross-host SSO — ADR-002 implementation; outposts on infra-services
• Authentik infra admin setup — provider, outpost, token, and verification walkthrough
• Komodo Authentik OIDC — native OIDC, not forward-auth
• Grafana Authentik auth proxy — forward-auth and auth proxy
• AdGuard edge SSO — disable UI auth behind forward-auth
• iOS apps and Authentik — Komodo Go API keys; OneUptime On-Call API route

Deploy and maintain

• Adding a service — onboard a new Compose stack
• Coordinated OS patching — Phase 8 weekly apt upgrades from infra-services
• Saltbox container updates — images and sb install on saltierpoop, not Komodo
• Central GitHub runner management — dispatcher matrix, offline runners, watchdog, qualification, and guarded maintenance
• Komodo explicit deployment — confirmed manual dispatch from main via self-hosted Actions
• Komodo image updates — native checks, reviewed draft pin PRs, and explicit cohort deployment
• Komodo connect LXC server — periphery and /opt/homelab for phoenix/sonarqube
• OneUptime upgrade and rollback — migration-safe release procedure
• infra-services capacity and resize — RAM/disk targets, Proxmox VM 123 resize
• Guest discovery — Proxmox/VMM reconciliation; operator SSH via sync-discovery-lxc-operator-keys.py
• Atuin sync — encrypted operator-shell history sync, account repair, and host setup

Network and devices

• HomeKit / AirPlay unreachable — check the phone VLAN, then Personal-to-IoT firewall access
• Roborock on IoT (VLAN 5) — DNS/ZBF/AdGuard when cloud vacuums go offline
• Network observations (2026-06-03) — dated scan anomalies and connectivity triage

Observability

• Monitoring dead-man — Healthchecks.io external heartbeat and ntfy
• OneUptime incident adapter — deploy, operate, and replay the race-safe alert adapter
• GitHub Actions runner observability — fleet, jobs, host metrics, logs, and alerts
• Central syslog (Graylog) — log routing and senders; Pattern E rollout recorded 2026-06-26
• LLM observability — LiteLLM gateway and Phoenix traces
• Prox memory soak — 24-hour attributed Prox RAM samples before shrink/retire
• Synology capacity ntfy — Whrrr volume alerts via ntfy.sh

Applications

• Order Ledger Google OAuth — Gmail client, SOPS merge, Connect Gmail
• JDownloader2 over Mullvad (Saltbox) — Gluetun and sandbox-jdownloader2 on saltierpoop
• qBittorrent over Mullvad (Saltbox) — Gluetun-shared qBittorrent, replacing qbittorrentvpn
• HAOS retention and Ford timeline — 15-month recorder and Mach-E map dashboard
• PostgreSQL Companion pgvector — deploy and verify pgvector for Companion, not tdler
• Telegram Companion owner E2E — lab slice for the Mac archive app; full checklist in Companion
• Nocturne preview drain — PC writer; public route via Traefik and UniFi 10011

Programs and migrations

Check each program's recorded status, owner decisions, and acceptance gates before proceeding.

• Operations platform migration — staged OneUptime rollout, gates, and rollback
• OneUptime v12 merge program — owner map for PRs 351–354
• PR 352 v12 platform — owner cutover: backup, merge, deploy-oneuptime
• PR 353 AI evidence — owner enablement; evidence off by default
• PR 354 remediation broker — owner canary VM, secrets, mutation lock
• Phase 7 owner actions — firewall, DSM, WiFi, Tailscale, AdGuard, PiHole decommission
• Phase 7R audit questionnaire — takeover audit owner Q&A dated 2026-06-18
• Phase 7R ZBF remediation — UDM policies from audit decisions

Documentation

• Documentation validation — verify docs match live state
• Cloudflare Pages (docs deploy) — MkDocs deployment to hldocs-c0acdec9.pages.dev

Historical reference

Retired systems, rejected pilots, and completed decommission records are not pending operating procedures. See History for the broader record.

• Wazuh SIEM (retired) — stack removed; Graylog remains the log path
• Wazuh edge SSO (retired) — stack and Authentik route removed
• Historical GitHub runner fleet — WSL-era fleet retired 2026-07-27; do not register it
• OneUptime runner incident pilot — rejected 2026-07-31; keep the native workflow disabled
• Decommission old monitoring — LXC 101/108/112 migration and teardown completed in May 2026
• Compute decommission queue — backup and destroy record completed 2026-06-25
• Metrimon decommission gate — VM 106 checklist passed; VM destroyed 2026-06-24