Skip to content

Disaster Recovery: Rebuild from Zero

How to rebuild the homelab from scratch using offsite tier-1 backups and this git repository. This runbook assumes total loss of all local infrastructure.

What You Need

  1. This git repository — clone from GitHub
  2. SOPS age private key — stored in 1Password (vault: homelab)
  3. B2 credentials — stored in 1Password (vault: homelab) or in the SOPS-encrypted infra/ansible/group_vars/all/backup.sops.yaml
  4. Restic repo password — stored in 1Password (vault: homelab) or in the SOPS-encrypted backup vars
  5. A fresh Ubuntu machine to use as the recovery staging host

Recovery Order

Recovery proceeds tier-1-first, infrastructure-before-applications:

1. Proxmox hypervisor (manual install)
2. infra-services VM (provision, bootstrap)
3. Restore tier-1 backups from B2
4. Traefik (reverse proxy — everything depends on this)
5. Komodo (stack reconciler)
6. Monitoring (Prometheus, Grafana, Alertmanager)
7. saltierpoop VM (Saltbox — manual reinstall, restore Authentik DB)
8. Remaining services (ARA, Homepage)
9. External services (Harbor, HAOS)

Step-by-Step

1. Provision the hypervisor

Install Proxmox VE on the ASUS MINIPC PN64. Default config is fine. After install, set a static IP at 192.168.6.71 on vmbr0.

2. Provision infra-services VM

Create an Ubuntu 24.04 VM on Proxmox (or Synology VMM if Whrrr survived):

  • 4 vCPU, 4-8GB RAM, 50GB disk
  • Static IP: 192.168.6.17
  • CPU type: host (for AVX support)

Install Docker:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

3. Clone the repo and bootstrap secrets

git clone https://github.com/notarealemail/homelab.git /opt/homelab
cd /opt/homelab

# Retrieve the age key from 1Password and place it
sudo mkdir -p /etc/homelab
# Copy age-key.txt from 1Password to /etc/homelab/age-key.txt
sudo chmod 600 /etc/homelab/age-key.txt

4. Install restic and restore tier-1 from B2

# Install restic
wget https://github.com/restic/restic/releases/download/v0.17.3/restic_0.17.3_linux_amd64.bz2
bunzip2 restic_0.17.3_linux_amd64.bz2
chmod +x restic_0.17.3_linux_amd64
sudo mv restic_0.17.3_linux_amd64 /usr/local/bin/restic

# Set up B2 credentials (from 1Password)
export B2_ACCOUNT_ID="..."
export B2_ACCOUNT_KEY="..."
export RESTIC_PASSWORD="..."  # restic repo password from 1Password

# List B2 repos
restic -r b2:your-bucket:komodo snapshots
restic -r b2:your-bucket:traefik snapshots

# Restore each
restic -r b2:your-bucket:traefik restore latest --target /tmp/restore-traefik
restic -r b2:your-bucket:komodo restore latest --target /tmp/restore-komodo

5. Bring up Traefik

cd /opt/homelab/services/traefik

# Decrypt secrets
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d .env.sops.yaml \
  | sed 's/: /=/' > .env

# Restore config from backup
cp -a /tmp/restore-traefik/opt/homelab/services/traefik/config/* ./config/
cp -a /tmp/restore-traefik/opt/homelab/services/traefik/acme.json ./acme.json 2>/dev/null || true
chmod 600 ./acme.json 2>/dev/null || true

docker compose up -d

6. Bring up Komodo

cd /opt/homelab/services/komodo

# Generate fresh secrets for compose.env (or restore from 1Password)
# See services/komodo/README.md for the list of GENERATE_ON_HOST values

# Restore MongoDB backups
sudo mkdir -p /etc/komodo/backups
cp -a /tmp/restore-komodo/etc/komodo/backups/* /etc/komodo/backups/

docker compose --env-file compose.env up -d

7. Bring up monitoring

cd /opt/homelab/services/monitoring

# Decrypt secrets
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d .env.sops.yaml \
  | sed 's/: /=/' > .env

docker compose up -d

8. Bring up remaining services

# ARA
cd /opt/homelab/services/ara && docker compose up -d

# Homepage
cd /opt/homelab/services/homepage && docker compose up -d

9. Re-provision saltierpoop (if needed)

Saltierpoop is a Saltbox-managed VM. Full rebuild:

  1. Create a fresh Ubuntu VM on Proxmox (20 vCPU, 45GB RAM, 260GB disk)
  2. Run the Saltbox installer: curl -sL https://install.saltbox.dev | bash
  3. Restore accounts.yml and settings.yml from SOPS — see command below
  4. Restore Authentik DB from B2 backup (if available)
  5. Run sb install saltbox to rebuild the full stack

Restore SOPS secrets:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt \
  sops -d /opt/homelab/infra/ansible/roles/saltbox-host/files/accounts.sops.yaml \
  > /opt/saltbox/accounts.yml

10. DNS and networking

  • Configure UDM SE with static routes and DNS policy table entries for *.infra.realemail.app pointing to 192.168.6.17
  • Re-add Cloudflare DNS records if the domain config was lost

Post-Recovery Verification

  • [ ] All Docker containers running: docker ps on infra-services
  • [ ] Traefik issuing valid certs: check https://homepage.infra.realemail.app
  • [ ] Prometheus scraping targets: check prometheus.infra.realemail.app/targets
  • [ ] Grafana dashboards loading: check grafana.infra.realemail.app
  • [ ] Komodo polling repo: check komodo.infra.realemail.app
  • [ ] Ansible-pull running on managed hosts: check ARA at ara.infra.realemail.app
  • [ ] Backup timers active: systemctl list-timers restic-*

What This Does NOT Cover

  • Tier-3 media (~290TB): deliberately not backed up offsite. Synology RAID is the only protection. If Whrrr is lost, media is gone.
  • Customer apps (recordurbate, ubuncap): separate infrastructure on Whrrr VMs. Not covered by this DR plan.
  • Network hardware config: UDM SE, switches, and APs retain their config internally. UniFi cloud backup or manual export recommended separately.

Key Contacts and Credentials

All credentials are in 1Password under the "homelab" vault:

  • age private key (SOPS decryption)
  • B2 application key (offsite backup access)
  • restic repository password
  • GitHub PAT (Komodo Git Provider)
  • Cloudflare API token (DNS-01 challenges)
  • Discord webhook URL (Alertmanager notifications)

See secrets.md for the full secrets inventory.