Skip to content

Per-Service Restore Procedures

How to recover each backed-up service from restic snapshots.

For active backup alerts, start with Backup alert triage to identify whether the failure is metrics collection, systemd scheduling, local snapshots, copy targets, or restore validation.

Prerequisites

  • restic installed on the recovery host
  • Access to /etc/restic/password (the restic repo encryption key)
  • For tier-1 offsite restores: B2 credentials in /etc/restic/b2.env

General Restore Pattern

All services follow the same basic flow:

# 1. List available snapshots
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service>
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots

# 2. Restore the latest (or a specific snapshot)
restic restore latest --target /tmp/restore-<service>
# or: restic restore <snapshot-id> --target /tmp/restore-<service>

# 3. Copy restored files to the service directory
# (service-specific steps below)

# 4. Restart the service
cd /opt/homelab/services/<service>
docker compose down && docker compose up -d

The runtime backup manifest lives beside each stack as services/<name>/backup.yml. This runbook records the restore steps that are not obvious from the manifest alone.

Service Tier Restore note
Komodo 1 Mongo dump directory
Traefik 1 Static config + ACME cert cache
OneUptime 1 PostgreSQL + ClickHouse logical backups and SOPS encryption key
Paperless 1 NFS media/inbox + data volume + Postgres dump + export
ARA 2 SQLite Docker volume
Homepage 2 Generated config; rerun generator if easier
AdGuard 2 AdGuard config/work volumes + Unbound config
Monitoring 2 Grafana custom data outside provisioning
LiteLLM 2 Postgres volume with spend logs and virtual keys
Stirling-PDF 2 Configs, tessdata, pipeline volumes
Phoenix 2 Phoenix Postgres/data volumes on LXC 124
Wazuh 2 Indexer, manager, Filebeat, dashboard assets, TLS material
Authentik outpost 3 Stateless; recreate from Authentik + SOPS token

Restoring from B2 (offsite, tier-1 only)

If the local repo is lost:

source /etc/restic/b2.env
export RESTIC_REPOSITORY=b2:<bucket>:<service>
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots
restic restore latest --target /tmp/restore-<service>

Komodo (tier 1)

Komodo auto-manages MongoDB dumps to /etc/komodo/backups. Restic backs up that directory.

# Restore
restic restore latest --target /tmp/restore-komodo

# Stop Komodo
cd /opt/homelab/services/komodo
docker compose down

# Replace the backups directory
cp -a /tmp/restore-komodo/etc/komodo/backups/* /etc/komodo/backups/

# Restart — Komodo will import the latest dump on startup
docker compose --env-file compose.env up -d

Verify: Open komodo.infra.realemail.app and confirm servers/stacks are listed. Check Komodo logs: docker logs komodo-core.

After any host Docker daemon restart, also recreate periphery so it remounts a live /var/run/docker.sock — otherwise deploys fail while the UI stays up. See komodo-periphery-docker-sock.md.

Traefik (tier 1)

Backs up the static config directory and acme.json (Let's Encrypt certs).

# Restore
restic restore latest --target /tmp/restore-traefik

# Stop Traefik
cd /opt/homelab/services/traefik
docker compose down

# Replace config
cp -a /tmp/restore-traefik/opt/homelab/services/traefik/config/* ./config/
cp -a /tmp/restore-traefik/opt/homelab/services/traefik/acme.json ./acme.json
chmod 600 ./acme.json

# Decrypt secrets
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt \
  sops -d --output-type dotenv .env.sops.yaml > .env

# Restart
docker compose up -d

Verify: curl -I https://homepage.infra.realemail.app returns a valid TLS cert. If acme.json was lost, Traefik re-issues certs automatically (may take a few minutes; watch for Let's Encrypt rate limits if you've done this multiple times in a week).

OneUptime (tier 1)

OneUptime requires a matched PostgreSQL dump, ClickHouse backup set, and SOPS ENCRYPTION_SECRET. Follow the dedicated OneUptime backup and restore runbook. Restore into isolated volumes first; start notification and remediation workflows only after incidents, timelines, encrypted values, and readiness are verified.

Paperless (tier 1)

Backs up Docker volumes (paperless_data, paperless_pgdata), Whrrr Scans NFS paths (/mnt/whrrr-scans/media, /mnt/whrrr-scans/inbox), the export/ bind mount, and a Postgres custom-format dump written by the backup pre-hook to services/paperless/backups/paperless.dump.

# Restore snapshot
restic restore latest --target /tmp/restore-paperless

cd /opt/homelab/services/paperless
docker compose down

# Restore volumes / NFS tree (adjust paths to match restic layout)
sudo cp -a /tmp/restore-paperless/var/lib/docker/volumes/paperless_data/_data/. \
  /var/lib/docker/volumes/paperless_data/_data/
sudo cp -a /tmp/restore-paperless/mnt/whrrr-scans/media/. /mnt/whrrr-scans/media/
sudo cp -a /tmp/restore-paperless/mnt/whrrr-scans/inbox/. /mnt/whrrr-scans/inbox/ || true
sudo cp -a /tmp/restore-paperless/var/lib/docker/volumes/paperless_pgdata/_data/. \
  /var/lib/docker/volumes/paperless_pgdata/_data/
sudo cp -a /tmp/restore-paperless/opt/homelab/services/paperless/export/. ./export/

# Decrypt secrets and start
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d --output-type dotenv .env.sops.yaml > .env
docker compose up -d

# If volume restore is incomplete, reload DB from dump instead:
# docker compose up -d db
# docker cp ./backups/paperless.dump paperless-db:/tmp/paperless.dump
# docker exec -i paperless-db pg_restore -U paperless -d paperless --clean --if-exists /tmp/paperless.dump
# docker compose up -d

Verify: Open https://paperless.infra.realemail.app, confirm documents and search. Optional logical check: docker exec paperless document_exporter /usr/src/paperless/export.

ARA (tier 2)

SQLite database with Ansible run history. Regeneratable but useful to keep.

restic restore latest --target /tmp/restore-ara

cd /opt/homelab/services/ara
docker compose down

# The volume is a Docker named volume; replace its contents
docker run --rm -v ara_ara-data:/data -v /tmp/restore-ara:/backup \
  alpine sh -c "rm -rf /data/* && cp -a /backup/* /data/"

docker compose up -d

Homepage (tier 2)

Config is generated from inventory — backup is convenience only.

restic restore latest --target /tmp/restore-homepage
# Copy config files back if needed. Or just re-run the generator:
cd /opt/homelab
uv run python inventory/generators/render-homepage.py

AdGuard (tier 2)

Backs up AdGuard config/work volumes plus unbound.conf.

restic restore latest --target /tmp/restore-adguard

cd /opt/homelab/services/adguard
docker compose down

docker run --rm \
  -v adguard_adguard-conf:/data \
  -v /tmp/restore-adguard/var/lib/docker/volumes/adguard_adguard-conf/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"

docker run --rm \
  -v adguard_adguard-work:/data \
  -v /tmp/restore-adguard/var/lib/docker/volumes/adguard_adguard-work/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"

cp -a /tmp/restore-adguard/opt/homelab/services/adguard/unbound.conf ./unbound.conf
docker compose up -d

Verify: clients can resolve through 192.168.6.17:53, AdGuard UI loads through Authentik, and generated rewrites still match inventory.

Monitoring (tier 2)

Grafana user customizations outside code-managed provisioning and the GitHub Actions collector SQLite state used for restart-safe event deduplication.

restic restore latest --target /tmp/restore-monitoring

# Replace Grafana data volume
cd /opt/homelab/services/monitoring
docker compose stop grafana github-actions-collector

docker run --rm \
  -v monitoring_grafana-data:/data \
  -v /tmp/restore-monitoring/var/lib/docker/volumes/monitoring_grafana-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/* /data/"

docker run --rm \
  -v monitoring_collector-data:/data \
  -v /tmp/restore-monitoring/var/lib/docker/volumes/monitoring_collector-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/* /data/"

docker compose --env-file .env up -d grafana github-actions-collector

Note: Dashboards provisioned from monitoring/grafana/dashboards/ are restored automatically on Grafana startup. Only user-created dashboards and annotations need the Grafana volume restore. If no collector backup exists, start it with an empty volume; the initial lookback repopulates state but can temporarily re-observe completed jobs.

LiteLLM (tier 2)

Restores the Postgres volume that stores spend logs, virtual keys, and key metadata. Provider API keys still come from SOPS-rendered .env.

restic restore latest --target /tmp/restore-litellm

cd /opt/homelab/services/litellm
docker compose down

docker run --rm \
  -v litellm_litellm-db-data:/data \
  -v /tmp/restore-litellm/var/lib/docker/volumes/litellm_litellm-db-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d --output-type dotenv .env.sops.yaml > .env
docker compose up -d

Verify: /v1/models returns HTTP 200, the Admin UI shows expected virtual keys, and the LiteLLM Grafana dashboard resumes metrics.

Stirling-PDF (tier 2)

Restores configs, OCR tessdata, and pipeline volumes. Uploaded PDFs are not retained by design.

restic restore latest --target /tmp/restore-stirling

cd /opt/homelab/services/stirling-pdf
docker compose down

for vol in configs tessdata pipeline; do
  docker run --rm \
    -v "stirling-pdf_${vol}:/data" \
    -v "/tmp/restore-stirling/var/lib/docker/volumes/stirling-pdf_${vol}/_data:/backup:ro" \
    alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"
done

docker compose up -d

Verify: Open https://stirling.infra.realemail.app (Authentik) and confirm saved pipelines/settings.

Phoenix (tier 2)

Phoenix runs on prox LXC 124, not infra-services. Restore on that host.

restic restore latest --target /tmp/restore-phoenix

cd /opt/homelab/services/phoenix
docker compose down

docker run --rm \
  -v phoenix_phoenix-db-data:/data \
  -v /tmp/restore-phoenix/var/lib/docker/volumes/phoenix_phoenix-db-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"

docker run --rm \
  -v phoenix_phoenix-data:/data \
  -v /tmp/restore-phoenix/var/lib/docker/volumes/phoenix_phoenix-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"

docker compose up -d

Verify: https://phoenix.infra.realemail.app loads through Authentik and LiteLLM's full trace test can query the litellm project.

Wazuh (tier 2)

Wazuh backups include indexer data, manager state, Filebeat state, dashboard custom assets, and generated TLS material.

restic restore latest --target /tmp/restore-wazuh

cd /opt/homelab/services/wazuh
docker compose --env-file compose.env down

restore_volume() {
  volume="$1"
  backup_path="$2"
  docker run --rm \
    -v "${volume}:/data" \
    -v "${backup_path}:/backup:ro" \
    alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"
}

restore_volume wazuh_wazuh-indexer-data \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-indexer-data/_data
restore_volume wazuh_wazuh-manager-data \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-manager-data/_data
restore_volume wazuh_wazuh-manager-etc \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-manager-etc/_data
restore_volume wazuh_wazuh-manager-var-multigroups \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-manager-var-multigroups/_data
restore_volume wazuh_wazuh-manager-integrations \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-manager-integrations/_data
restore_volume wazuh_wazuh-manager-active-response \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-manager-active-response/_data
restore_volume wazuh_wazuh-manager-agentless \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-manager-agentless/_data
restore_volume wazuh_wazuh-manager-wodles \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-manager-wodles/_data
restore_volume wazuh_wazuh-filebeat-etc \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-filebeat-etc/_data
restore_volume wazuh_wazuh-dashboard-custom \
  /tmp/restore-wazuh/var/lib/docker/volumes/wazuh_wazuh-dashboard-custom/_data

cp -a \
  /tmp/restore-wazuh/opt/homelab/services/wazuh/config/wazuh_indexer_ssl_certs \
  ./config/

bash scripts/sync-wazuh-yml.sh
docker compose --env-file compose.env up -d
bash scripts/apply-security-config.sh
docker compose --env-file compose.env ps

Verify: wazuh-indexer, wazuh-manager, and wazuh-dashboard are healthy; the dashboard loads through Authentik; agents return to Active as they reconnect.

Authentik outpost (tier 3)

The infra-services outpost is stateless. Its token is SOPS-managed, and the authoritative provider/outpost state lives in the Authentik server on saltierpoop.

cd /opt/homelab/services/authentik-outpost
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d .env.sops.yaml \
  | sed 's/: /=/' > .env
docker compose up -d

If the token is invalid or lost, recreate it from the Authentik admin flow in authentik-infra-admin-setup.md, update .env.sops.yaml, and redeploy the outpost.

External Services (not on infra-services)

Saltbox Secrets (saltierpoop)

The SOPS-encrypted source files are in git. To restore the deployed copies:

cd /opt/homelab
SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt \
  sops -d infra/ansible/roles/saltbox-host/files/accounts.sops.yaml \
  > /opt/saltbox/accounts.yml

Or re-run Ansible: ansible-pull will deploy the latest encrypted versions.

Authentik DB (saltierpoop)

Restore from the pg_dump backup:

restic restore latest --target /tmp/restore-authentik

docker exec -i authentik-postgres psql -U authentik \
  < /tmp/restore-authentik/tmp/authentik-backup/authentik.sql.gz

docker restart authentik

HAOS Config

Full snapshots (~2 GiB). NAS (/mnt/infra-backups/homelab-restic) is the primary restic repo; B2 is a safety net only (keep-last 2 on B2; weekly full + daily incrementals). Local staging under /var/backups/external-staging is cleared after a successful backup.

From NAS (primary):

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/external-haos-config
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots
restic restore latest --target /tmp/restore-haos

From B2 (total-loss / offsite DR):

source /etc/restic/b2.env
export RESTIC_REPOSITORY=b2:<bucket>:external-haos-config
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots
restic restore latest --target /tmp/restore-haos

Restore the snapshot .tar via the HA Supervisor API or manually upload through the HAOS UI at Settings > System > Backups.