Skip to content

Disaster recovery: rebuild from zero

Recovery order for loss of the lab's local infrastructure. This is a coordinated rebuild guide; the per-service restore procedures contain the data restore steps. A complete loss-of-lab rehearsal was not performed in the 2026-10-24 documentation audit.

For a limited TLS or authentication outage, use public edge recovery.

Recovery inputs

  • A reviewed revision of this private repository and the current inventory.
  • The operator SOPS age key, restic password, B2 credentials and DNS credentials recovered through secrets management.
  • A Linux recovery machine with working DNS/internet, Git, SOPS, restic on PATH, Python 3 with PyYAML (python3-yaml host package), and Docker.
  • A root Bash shell, the reviewed checkout at /opt/homelab, and both RESTIC_REPOSITORY and RESTIC_PASSWORD_FILE exported for each dataset.
  • Confirmed snapshot IDs and destination storage. Inventory is the target-state reference; dated discovery captures and old plan sizing are not sizing advice.

Keep recovered credentials out of command history, logs and documentation. Use the service-specific renderers to rebuild runtime files from encrypted source.

Recovery order and gates

Stage Work Gate before continuing
1 Recover network management, VLANs, routing and administrative SSH Reach intended recovery hosts without bypassing the authentication design
2 Provision Proxmox, the infra-services VM, and necessary storage Hostnames/IPs match reviewed inventory; storage has room for staging and live data
3 Restore DNS and recover NAS mounts or replacement storage AdGuard resolves internal names; mount sources exist before dependent containers start
4 Recover Saltbox and its Authentik database Auth server is available before validating OIDC and outposts
5 Recover Traefik, outposts and Komodo TLS, Authentik and explicit Komodo database restore verified
6 Recover PostgreSQL LXC/application databases and other Tier-1 datasets Isolated restore checks pass before attaching applications
7 Restore monitoring and incident state Scrapes, logs and incident correlation verified; notification/remediation gates reviewed
8 Restore remaining services and backup scheduling Application smoke tests pass; new snapshots and offsite copies succeed

Emergency SSH and direct recovery tools may be needed while DNS/SSO is down; that does not authorize removing production authentication gates.

Provision hosts from current inventory

Use prox and infra-services, not the original PLAN.md guest sizes. At the audited revision, infra-services is specified as 12 vCPU, 28 GiB RAM and 200 GiB disk with CPU type host. The live hypervisor reports 8 vCPU and 28 GiB RAM for VM 123; reconcile that difference with the operator before sizing a replacement. The former 4–8 GiB RAM / 50 GiB disk example cannot represent the current workload. Proxmox is Debian 13/PVE 9 in the live audit; its inventory OS version still needs reconciliation as recorded in the validation journal.

Bootstrap host access, the encrypted secret key, Docker and the required bind mounts before restoring applications. Use the SSH runbook and service onboarding. Saltbox owns its own container lifecycle.

Retrieve and validate snapshots

Use the guarded staging contract for every dataset. The producer change in PR #469 creates one snapshot containing all manifest paths, tagged homelab-path-set-v1. This is source work, not a deployed or drilled recovery guarantee. Old snapshots remain split; neither the tag nor complete root paths prove a valid backup or application consistency.

For each dataset, inspect services/<name>/backup.yml or backups/external/<name>.yaml for the repository name and captured paths. backups/policies.yaml describes tiers; it is not the runtime job inventory. Use the absolute manifest path regardless of the current working directory. For example, stage the offsite Authentik dataset before recovering identity:

# Root Bash on the recovery host; load credentials via secrets recovery.
umask 077
source /etc/restic/b2.env
read -r -p "Recovery B2 bucket name: " B2_BUCKET
export RESTIC_PASSWORD_FILE=/etc/restic/password
export RESTIC_REPOSITORY="b2:${B2_BUCKET}:external-authentik-db"
restic snapshots --json || exit 1
read -r -p "Reviewed offsite snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/authentik-db.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-authentik || exit 1

The default is metadata/preflight JSON only. Review the ID, host, time and exact roots. /tmp/restore-authentik must be absent, with existing non-symlink parents; do not pre-create it. Then stage into the new private mode-0700 directory:

python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/authentik-db.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-authentik --restore || exit 1

Preserve any failed staging and use a new absent target for retries. For legacy split jobs, repeat --snapshot only for reviewed disjoint same-host snapshots whose combined roots exactly match the manifest; assess their consistency separately. --latest is for scheduled verification only: the newest snapshot must be complete, with no older-snapshot fallback or automatic mixing.

Do not stop services, delete data, or transfer/copy staged data until all staging for the recovery step has been validated. Run application-specific isolated drills and review checksums, matching secrets and database versions before seeking owner approval for cutover. The helper performs no cutover, promotion or cleanup. Remaining application-specific recovery defects in the procedures below are not certified or fixed by this path-set repair. Restoring database files into a running database is not an isolated validation.

Retain legacy backups during migration. Restic's default host,paths retention groups keep old split path groups separate from the new complete set. This repair authorizes no automatic deletion or retention-group change; check storage and obtain cleanup approval only after verifying new backups and application restores.

Restore identity, ingress and deployment

  1. Recover Saltbox's encrypted accounts.yml and settings.yml through the saltbox-host role. Their current destination is /srv/git/saltbox/.
  2. Restore Authentik from the external-authentik-db dataset. It contains a compressed cluster SQL dump; use the external restore procedure.
  3. Restore Traefik config and its named certificate volume using Traefik restore. acme.json is in traefik_traefik-certs, not a bind-mounted file beside Compose.
  4. Recover the infra Authentik outpost and test forward authentication and OIDC.
  5. Render Komodo's compose.env with scripts/render-komodo-compose-env.sh. Restore its database explicitly using Komodo restore. Starting Core with a backup directory does not import that backup.
  6. Verify Periphery bind mounts, repository access and ResourceSync before enabling deployment procedures. Follow the Komodo recovery context.

Restore data and applications

The restore index lists service-specific manifests and procedures. Recover shared PostgreSQL before dependent apps such as Reddit Explorer. Recover OneUptime, the incident adapter and the broker from their own logical backups; a running UI alone does not prove their authoritative state was restored.

Restore NAS-backed Paperless data and Reddit Explorer mounts before starting containers that use those directories. A missing NFS mount must not silently turn into writes to an empty local mountpoint. Review the service directory for source-specific prerequisites.

Do not deploy retired Wazuh or the retired Recordurbate customer application as part of a blanket recovery. Keep optional evidence collection and remediation mutation disabled until their separate acceptance gates are met.

Restore monitoring and resume automation

Render monitoring secrets with the maintained renderer; it also produces mounted secret files that a simple YAML-to-dotenv conversion would miss:

cd /opt/homelab
sudo bash scripts/render-monitoring-secrets.sh
cd services/monitoring
docker compose --env-file .env up -d

Verify alert triage, runner observability and external dead-man monitoring. Resume patching, export timers, deployment triggers and notifications only after their targets and credentials have been verified.

Acceptance record

Record the revision, snapshot IDs, recovered hosts, isolated drill results and remaining gaps in a dated journal entry. Required checks include:

  • DNS, TLS, Authentik login/OIDC and operator access.
  • Application data counts and health, not just container startup.
  • Komodo resource reconciliation without unintended deployments.
  • Prometheus targets, logs, incident adapter readiness and backup alerts.
  • New backup snapshots, offsite copy results and restore-test scheduling.
  • Managed-host pull/patch timers and runner routing.

Tier-3 media has no offsite recovery guarantee. Network device backups, physical hardware replacement, and application-team-owned workloads need their own recovery inputs; this repository cannot recreate missing data.