Skip to content

Per-Service Restore Procedures

How to recover each backed-up service from restic snapshots.

For active backup alerts, start with Backup alert triage to identify whether the failure is metrics collection, systemd scheduling, local snapshots, copy targets, or restore validation.

Prerequisites

  • Run the snippets in a root Bash shell on the Linux recovery host, normally infra-services; Phoenix uses its own host. Start with umask 077.
  • Use a reviewed checkout at /opt/homelab, Python 3 with PyYAML (host package python3-yaml), and restic on PATH.
  • Set both RESTIC_REPOSITORY and RESTIC_PASSWORD_FILE in that shell. Access to /etc/restic/password is required; do not print credentials.
  • For offsite restores, recover B2 credentials in /etc/restic/b2.env.
  • Secret rendering requires SOPS and the safe rendering contract, including supported string values, required keys and trusted directory writers. The default automation key remains root-only; render and Compose use this same root context. Failed rendering must stop recovery before any subsequent deploy. Source tests do not establish live key access, client compatibility or recovery.
  • Each /tmp/restore-SERVICE target below must be absent, not an empty directory. Its parents must already exist without symlinks. Do not pre-create the target, reuse a previous attempt, or delete evidence to make the example work. If it exists, choose a new absolute target and update every downstream reference.

General Restore Pattern

The first recovery-contract repair (REC-001 / INC-002 / OPS-043) is source work in PR #469, not a deployment or restore-drill result. The new producer uses one restic invocation for all manifest paths, tagged homelab-path-set-v1. Older snapshots remain split. The tag describes a path set, not application consistency; unreadable files, exclusions, live database writes, and stale logical exports still need review.

Use backups/restore-snapshot.py to plan and stage recovery:

  1. Review restic snapshots --json for the intended repository, host, times and roots. Select full snapshot IDs, not an implicit newest snapshot.
  2. Run the helper without --restore: it reads metadata and preflights the target, then emits a JSON plan. It does not create the target or restore data.
  3. Review the plan; repeat the same arguments with --restore. The helper creates a private mode-0700 target, restores there only, and checks expected roots.
  4. Validate all staged paths and the application recovery set before any service stop, deletion, transfer, or copy into live storage. Preserve staging and failure evidence; a retry requires a new absent target.
  5. Only a separately reviewed, owner-approved application recovery plan may authorize cutover. The helper performs no hooks, promotion, cleanup or live copy.

Safety gate: root-path completeness is not proof of a valid backup. The application-specific commands below are retained recovery context, not certified by this path-set repair. Remaining application recovery defects are not fixed here. Review exact paths, database versions, logical imports, checksums, secrets, mounts and isolated drill results before using those commands. Do not run a page end-to-end as a script or stop/delete/copy anything merely because staging passed.

The runtime manifest is services/<name>/backup.yml, or the appropriate backups/external/*.yaml for an external dataset. Snippets use absolute manifest paths so a previous cd cannot select the wrong manifest. The helper requires selected roots to exactly match the manifest: no missing, extra, repeated or ancestor/descendant-overlapping roots.

For legacy split snapshots, explicitly repeat --snapshot for the reviewed, disjoint roots from the same hostname. Their union must exactly match the manifest. Record the times and assess cross-snapshot consistency yourself; the helper does not establish a shared point in time. For example, after reviewing OneUptime's two legacy path snapshots, plan only:

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/oneuptime
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json
read -r -p "Reviewed PostgreSQL-path snapshot ID: " POSTGRES_SNAPSHOT_ID
read -r -p "Reviewed ClickHouse-path snapshot ID: " CLICKHOUSE_SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/oneuptime/backup.yml \
  --snapshot "$POSTGRES_SNAPSHOT_ID" --snapshot "$CLICKHOUSE_SNAPSHOT_ID" \
  --target /tmp/restore-oneuptime

Use the same two IDs with --restore only after reviewing the plan. Do not combine snapshots whose paths overlap, even if their times appear compatible. If a current manifest differs from the historical roots, stop and review the historical recovery contract rather than weakening the manifest merely to pass preflight.

--latest is scheduled verification only: the globally newest snapshot must be unique and complete by itself. There is no fallback to an older complete snapshot and no automatic assembly of split snapshots. Newest-time ties fail.

Restic's default retention grouping is host,paths. Legacy split path groups remain separate from the new complete-path group; new backups do not automatically retire those old groups. No automatic deletion or retention-group change is authorized by this repair. Check local/offsite capacity, verify new backups and application restores, then seek separate approval for any cleanup.

Service Tier Restore note
Komodo 1 Mongo dump directory
Traefik 1 Static config + ACME cert cache
OneUptime 1 PostgreSQL + ClickHouse logical backups and SOPS encryption key
PostgreSQL Companion (external) 1 companion, tdler, reddit_explorer, globals, and row-count evidence
Paperless 1 PostgreSQL 16 logical generations + data/NFS media/inbox/export; database smoke is not full recovery
Firefly III 1 Postgres dump + firefly_pgdata / firefly_upload volumes
Rexit 1 Private archive with manifest, SQLite, gallery, and media validation
OneUptime incident adapter 1 Authoritative correlation DB; adapter recovery
OneUptime remediation broker 1 Audit, reservations and kill switch; broker runbook
Order Ledger 1 Consistent SQLite snapshot; see section below
Reddit Explorer mixed PostgreSQL via external cluster backup; media/stills are regeneratable; service guide
SonarQube 2 Dedicated host volumes; host guide
Paperless–Stirling Bridge 2 Persistent document-ID audit history; application and pipelines from Git
PR Dashboard 3 Replaceable state; service guide
Monitoring canary 3 Stateless; service guide
ARA 2 SQLite Docker volume
Homepage 2 Generated config; rerun generator if easier
AdGuard 2 adguard-conf only; no work/query history. Archived unbound.conf is not mounted by Compose
Monitoring 2 Grafana custom data outside provisioning
LiteLLM 2 PostgreSQL 16 logical generations for spend logs and key metadata; historical raw PGDATA is unproven
Stirling-PDF 2 Configs, tessdata, pipeline volumes
Phoenix 2 Phoenix Postgres/data volumes on LXC 124
Wazuh retired Do not restore. Historical snapshots stay in the existing repositories
Authentik outpost 3 Stateless; recreate from Authentik + SOPS token

Restoring from B2 (offsite-enabled jobs)

If the local repo is lost, use the same manifest and staging checks against the B2 repository. Tier 1 normally has offsite copies; HAOS is an explicitly enabled tier-2 exception. Repository names for external jobs include external-, but manifest filenames do not. This example plans an external Authentik restore:

source /etc/restic/b2.env
read -r -p "Recovery B2 bucket name: " B2_BUCKET
export RESTIC_REPOSITORY="b2:${B2_BUCKET}:external-authentik-db"
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json
read -r -p "Reviewed offsite snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/authentik-db.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-authentik || exit 1

After reviewing the JSON plan, stage only:

python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/authentik-db.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-authentik --restore || exit 1

Choose IDs from the offsite repository itself, not assumed local IDs. Validate all staging before following the application recovery context below.

Komodo (tier 1)

Restic stores Komodo's dated database backups from /etc/komodo/backups. Copying those files back and starting Core does not import the database. Use the explicit Komodo CLI restore operation against a reviewed target database.

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/komodo
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/komodo/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-komodo || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/komodo/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-komodo --restore || exit 1

Select the dated backup directory inside the restored etc/komodo/backups/ tree. Follow Komodo's database restore procedure: run km database restore with a /backups mount and the explicit KOMODO_CLI_DATABASE_TARGET_* settings. Restore into an isolated empty database first and verify resources and users. Restoring into a populated database can retain old records; it is not an implicit replacement operation.

The CLI restore and a full Komodo recovery were not executed during the 2026-10-24 documentation audit. Do not treat this source review as a restore drill. Only after all staging and the isolated database restore are validated may the owner approve stopping Core/Periphery and selecting the recovered database. These application steps are not certified by the path-set repair. After that separate cutover, start Core and Periphery:

set -euo pipefail
cd /opt/homelab/services/komodo || exit 1
docker compose --env-file compose.env up -d

Verify OIDC login, servers, stacks and ResourceSync before re-enabling deployment procedures. After a host Docker restart, recreate Periphery as described in the stale socket runbook.

Traefik (tier 1)

Backs up the static config directory and acme.json (Let's Encrypt certs).

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/traefik
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/traefik/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-traefik || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/traefik/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-traefik --restore || exit 1

Stop here: validate all staged config and certificate data, then obtain application cutover approval. The following live steps are not certified by this repair.

set -euo pipefail
# Stop Traefik only after the validation and approval gate
cd /opt/homelab/services/traefik || exit 1
# Render preflight BEFORE any down/config/certificate mutation; stop on failure.
python3 /opt/homelab/scripts/render-sops-env.py \
  --source .env.sops.yaml --output .env \
  --rename CF_DNS_API_TOKEN=CLOUDFLARE_DNS_API_TOKEN \
  --require CLOUDFLARE_DNS_API_TOKEN || exit 1

docker compose down

# Replace config
cp -a /tmp/restore-traefik/opt/homelab/services/traefik/config/* ./config/
# The certificate cache is a named volume, not ./acme.json.
docker volume create traefik_traefik-certs
docker run --rm -v traefik_traefik-certs:/certs \
  -v /tmp/restore-traefik/var/lib/docker/volumes/traefik_traefik-certs/_data:/backup:ro \
  alpine:3.22 sh -c 'cp -a /backup/. /certs/ && chmod 600 /certs/acme.json'

# Restart
docker compose up -d

Verify: curl -I https://homepage.infra.realemail.app returns a valid TLS cert. If acme.json was lost, Traefik re-issues certs automatically (may take a few minutes; watch for Let's Encrypt rate limits if you've done this multiple times in a week).

OneUptime (tier 1)

OneUptime requires a matched PostgreSQL dump, ClickHouse backup set, and SOPS ENCRYPTION_SECRET. Follow the dedicated OneUptime backup and restore runbook. Restore into isolated volumes first; start notification and remediation workflows only after incidents, timelines, encrypted values, and readiness are verified.

Rexit (tier 1)

Validate a snapshot in isolation before replacing the live private archive:

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/rexit
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/rexit/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-rexit || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/rexit/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-rexit --restore || exit 1
python3 /opt/homelab/backups/restore-drills/rexit.py \
  /tmp/restore-rexit/srv/rexit

Only after all staging and the archive drill validate, and the owner approves cutover, consider these application steps; the path-set repair does not certify them:

set -euo pipefail
cd /opt/homelab/services/rexit || exit 1
sudo systemctl stop rexit-export.timer
sudo systemctl stop rexit-export.service
docker compose stop studio

sudo rsync -a --delete /tmp/restore-rexit/srv/rexit/ /srv/rexit/
sudo chown -R 65532:65532 /srv/rexit

docker compose up -d studio
sudo systemctl start rexit-export.timer

Confirm the Authentik-protected gallery loads, then run systemctl start rexit-export.service only if an immediate incremental refresh is wanted. For total local-repository loss, use the B2 procedure above with the rexit repository.

Paperless (tier 1)

REC-004 in PR #474 replaces hot PGDATA backup/recovery with PostgreSQL 16 logical generations. This is a source contract, not evidence of deployment or a successful application restore. The manifest retains paperless_data, /mnt/whrrr-scans/media, /mnt/whrrr-scans/inbox, /opt/homelab/services/paperless/export and the untouched legacy /opt/homelab/services/paperless/backups directory. New logical generations live under /var/backups/postgres-logical/paperless, outside the shared checkout; new snapshots no longer include paperless_pgdata. Tier 1 and B2 offsite stay unchanged.

The shared producer defaults to an offline plan with --service paperless; the scheduled pre-hook explicitly invokes python3 with --backup and DOCKER_HOST=unix:///run/docker.sock. Each completed private, immutable generation under /var/backups/postgres-logical/paperless contains paperless.dump, paperless.dump.sha256 and manifest.json. Metadata binds the service, database/user, PostgreSQL server/client major 16, capture time, filename, byte count and SHA-256. When supplied, a validated systemd INVOCATION_ID is bound as invocation_id.

Exports require root. Backup-client's logical-postgres.yml provisions the namespace and service directories as root:root mode 0700 before deploying units; foreign-owned namespaces and symlinks are refused. The producer requires those parents to exist and never changes existing directory permissions. Shared checkout group-write normalization is why new generations cannot live beside the legacy dump. The producer, pruner and verifier require Python 3.11+; the producer and pruner also need host PyYAML. Normal role apply must ensure Python 3/PyYAML and pass a read-only version/import check before units; check mode cannot certify missing dependencies. Require full backup convergence with mounts, repositories and credentials ready; a systemd-only refresh is valid only after those base prerequisites, not as bootstrap. The source host's observed Python 3.12.3 is prerequisite metadata, not database-query, export or recovery evidence.

Bounded storage and scheduled retention

The bounded-storage follow-up in PR #474 enforces local storage limits, superseding the initial PR's unbounded retention and lack of headroom enforcement. Owner policy review is required before merge and scheduling: merge/convergence now enables future scheduled local deletion. No live operation or deployment acceptance is claimed here.

The fixed storage policy retains seven newest validated completed generations per service, by generation ordering, not seven days; retries can shorten local time coverage. At 14 generations, new exports refuse without deleting anything. Paperless custom dumps are capped at 64 MiB (LiteLLM 512 MiB), catalog scratch at 32 MiB and each checksum/manifest at 64 KiB. Before Docker queries or staging writes, preflight budgets the maximum dump + catalog + 2 × 64 KiB metadata and 5 inodes.

Available bytes must meet max(ceil(15% of total bytes), 8 GiB) + 16 GiB plus prospective bytes; available inodes must meet max(ceil(5% of total inodes), 100,000) plus prospective inodes. Checks recur during streaming and allocation; unavailable or invalid accounting refuses writes. The shared lock serializes cooperating producers/pruners only: no preallocation or quota constrains unrelated writers. Failed .staging-*, incomplete .deleting-*, unknown/foreign/corrupt/oversize entries and unsafe links block new exports and pruning, never automatic deletion or repair. Both private marker patterns are excluded from restic. Preserve evidence and request review; do not remove it or override limits to force a capacity pass.

The dedicated pruner with --service paperless defaults to a read-only filesystem plan of the actual namespace, without Docker/systemd calls or writes. It is not the producer's offline policy plan, a capacity certificate or a standalone operator cleanup recipe. Explicit --prune is only for the direct Python systemd ExecStartPost, after successful NAS backup/forget and required Paperless B2 copy/forget. It requires matching INVOCATION_ID/unit InvocationID, this process as current ControlPID, activating/start-post, and exactly one matching manifest generation that is the newest. All generations are validated before any deletion; manual exports without invocation binding cannot authorize cleanup. Earlier required failures prevent pruning. Role gates require boolean local_generation_retention: true, no fetch/post-hook or success-code overrides, and Paperless offsite: true with a nonempty B2 bucket even in check mode. Reviewed unit ordering and successful exit codes do not prove payload completeness or offsite/application recovery.

Across both services the hard-ceiling dump payload is 14 × 576 MiB = 7.875 GiB, plus bounded metadata/catalog and filesystem overhead. Restic retention and historical host,paths groups are unchanged; local retention does not remove dump bytes already in snapshots or add automatic legacy-data/historical-snapshot cleanup. The full operating contract records exact accounting and authorization boundaries.

PostgresLogicalBackupRunFailed is a new critical / page alert only for the Paperless/LiteLLM jobs on infra-services, including pre-hook and NAS/B2/pruner failures. It uses the existing page route; source configuration is not delivery acceptance, and OPS-042 remains open. Follow the failure triage, not a forced cleanup or repeated-export workaround.

Stage a reviewed recovery set

Use the prerequisites above, Python 3.11+ and a full reviewed checkout, including the shared Python modules; copying just the service or one helper is insufficient. Recover restic/offsite credentials separately from application secrets. These examples stage data only; they neither decrypt/render .env nor authorize service changes.

Budget the complete snapshot, not seven dumps: latest snapshots can contain 14 generations before post-backup pruning. Paperless staging can duplicate up to 896 MiB of generation payload plus metadata/filesystem overhead and all other captured roots (data, media, inbox, export, legacy backups), alongside live data and verifier memory/tmpfs. The operational reserve is not a restore-capacity guarantee.

Historical roots differ: the commands below use the new manifest. For an older snapshot containing PGDATA, supply a separately reviewed historical manifest to both guarded staging calls instead. For legacy split snapshots, explicitly select all reviewed same-host, disjoint IDs whose union matches that historical manifest. Do not edit/weaken the new manifest, omit roots, or treat a staged hot PGDATA tree as a proven physical backup. Preserve historical snapshots and evidence.

umask 077
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/paperless
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed full snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/paperless/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-paperless || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/paperless/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-paperless --restore || exit 1

Validate all staged data, media, inbox and export roots and their recovery-time relationship to the selected database. Select a completed generation explicitly, not whichever dump is newest or happens to exist in the checkout. Keep its dump, sidecar and manifest together, private, stable and free of symlink traversal.

Plan and run an isolated database smoke test

The verifier checks the selected staged artifact, not ./backups/paperless.dump in the current working directory. Without --verify it only validates local files and emits a plan: no Docker calls, image pull, database import or mutation. Only custom-format PGDMP archives are accepted. The owner must vet and separately preload an exact PostgreSQL 16 name@sha256:digest image before explicit verification; do not substitute a mutable tag or invented digest. The image must support the verifier's postgres user, standard entrypoint and PG16 layout; see the linked contract for the image checks.

read -r -p "Absolute staged generation path ending in /paperless.dump: " PAPERLESS_DUMP
case "$PAPERLESS_DUMP" in
  /tmp/restore-paperless/var/backups/postgres-logical/paperless/*/paperless.dump) ;;
  *) echo "Stop: select a generation from the reviewed staging target" >&2; exit 1 ;;
esac
read -r -p "Vetted preloaded PostgreSQL 16 image (name@sha256:digest): " POSTGRES_IMAGE
python3 /opt/homelab/backups/verify-postgres-backup.py \
  --service paperless --dump "$PAPERLESS_DUMP" --image "$POSTGRES_IMAGE" || exit 1

After reviewing the plan, approve the trusted local daemon and memory/tmpfs capacity. Require an explicit DOCKER_HOST=unix:///absolute/socket with safe ownership and no symlink traversal; review and unset conflicting Docker context/TLS overrides. The full endpoint and image requirements are in the logical backup contract linked above. Only then run:

read -r -p "Type VERIFY to approve a disposable database smoke test: " VERIFY_APPROVAL
test "$VERIFY_APPROVAL" = VERIFY || exit 1
python3 /opt/homelab/backups/verify-postgres-backup.py \
  --service paperless --dump "$PAPERLESS_DUMP" --image "$POSTGRES_IMAGE" --verify || exit 1

Explicit verification uses a fresh owned PostgreSQL 16 container with fixed 1 GiB data tmpfs / 2 GiB memory, --network=none, no published ports, no host data mounts and no application writers. Even dumps within the producer limit can expand beyond these bounds: the fixed 1 GiB smoke is not capacity certification and there is no sizing CLI override. It validates the archive/catalog and imports with pg_restore --exit-on-error --single-transaction --no-owner --no-acl, then requires a positive ordinary application-table count. Expected rows and original grants are not checked. Cleanup is limited to that exact owned temporary container, never backups, staged artifacts or live volumes. A passing result is a database smoke test, not full Paperless recovery. The older weekly generic file-count result was not application-restore certification; this change adds no automatic verifier wiring.

Historical fixed dump

An old backups/paperless.dump without generation metadata may use the verifier's legacy mode only with an independently trusted expected SHA-256. For example, derive it from the exact reviewed file streamed directly from its authenticated restic snapshot, with pipeline error checking and recorded snapshot/path provenance. Never hash the unknown staged copy to bless it, manufacture a manifest/sidecar, or bypass existing invalid metadata. Review the PostgreSQL version and archive too.

The historical file stages at /tmp/restore-paperless/opt/homelab/services/paperless/backups/paperless.dump, but inherited group-write permissions or foreign ownership can make the verifier reject that tree. Do not change permissions/ownership in the original staging. Obtain separate owner approval to copy the dump and all existing adjacent metadata into a new root-owned private validation directory outside the checkout and live/ backup roots, with trusted, non-symlink parents. Use a root:root mode-0700 directory and new owner-only regular files; preserve the original tree, bytes, ownership/modes and provenance as evidence. Copy bytes unchanged, not via hard links or symlink traversal; record the reviewed source-to-copy mapping privately.

Every existing manifest.json and .sha256 must accompany the copy unchanged, even if invalid. Never omit unreadable/invalid metadata, remove a manifest to force legacy mode, or compute the expected digest from the untrusted copy. Stop on unsupported layouts or incomplete copies. If a manifest exists, use the normal manifest-bound verifier without the legacy override and let invalid metadata fail. Only if the reviewed original truly lacked a manifest may the fixed historical dump use this filesystem-only legacy plan, selecting the approved private copy explicitly:

read -r -p "Absolute paperless.dump path in the approved private validation copy: " LEGACY_PAPERLESS_DUMP
read -r -p "Independently trusted expected SHA-256 (64 hex characters): " LEGACY_SHA256
read -r -p "Vetted preloaded PostgreSQL 16 image (name@sha256:digest): " POSTGRES_IMAGE
python3 /opt/homelab/backups/verify-postgres-backup.py \
  --service paperless --dump "$LEGACY_PAPERLESS_DUMP" \
  --image "$POSTGRES_IMAGE" --legacy-expected-sha256 "$LEGACY_SHA256" || exit 1

Only after the same owner/image/daemon/capacity gates may that exact legacy command be repeated with --verify. Missing trustworthy provenance is a stop condition, not permission to fall back to raw PGDATA.

Separate production cutover gate

Require an owner-approved plan for a fresh persistent database, matching PostgreSQL/application versions, private secret recovery/rendering, data/media/inbox and export reconciliation, durable Compose/storage/backup mappings, and rollback. The disposable verifier database is not a production target. Preserve original volumes, NFS data, snapshots and failed staging; do not clear or overwrite them.

Keep all application writers off until successful import and checks: Paperless web/consumer/background tasks, Paperless-GPT, Stirling pipelines, the Paperless–Stirling bridge and other scanner/API/NFS writers. Account for restart/deployment automation. Do not use a blanket Compose start before import, blind database deletion, or a raw-volume/start-and-then-import fallback. NFS mount/ownership checks and a coherent database/media/inbox recovery point remain required; REC-033 is still open.

Before reopening writers, complete the approved isolated application checks for expected document counts, originals/thumbnails/downloads, search and permissions, plus the rollback rehearsal. Restic access, SOPS secret recovery/rendering, database smoke, application integrity and production approval remain separate gates. Real PostgreSQL 16 export/import, application/media acceptance, rollback rehearsal and required offsite recovery remain open; source validation does not close them.

Firefly III (tier 1)

Backs up a Postgres custom-format dump written by the backup pre-hook to /opt/homelab/services/firefly/backups/firefly.dump (user/db firefly), plus Docker volumes firefly_pgdata (Postgres 18 data at /var/lib/postgresql) and firefly_upload. App URL is https://firefly.infra.realemail.app; the data importer is at https://firefly-import.infra.realemail.app. Containers: firefly, firefly-db, firefly-cron, firefly-importer.

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/firefly
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/firefly/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-firefly || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/firefly/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-firefly --restore || exit 1

Stop here: validate every staged root and an isolated logical database restore. The following database replacement and attachment steps need separate owner review; this repair does not certify them or authorize destructive SQL.

set -euo pipefail
cd /opt/homelab/services/firefly || exit 1
# Explicit manual opt-in; preflights Python/shared helper before secret access.
# Default automated Firefly consumers retain the shell/SOPS backend.
# Do not stop or start after render failure.
bash /opt/homelab/services/firefly/scripts/render-env.sh --restricted || exit 1
docker compose down

# Service name is db; container name is firefly-db
docker compose up -d db
# Wait until healthy, then ensure an empty target database if needed:
# docker exec -i firefly-db psql -U firefly -d postgres -c \
#   "DROP DATABASE IF EXISTS firefly; CREATE DATABASE firefly OWNER firefly;"

docker cp /tmp/restore-firefly/opt/homelab/services/firefly/backups/firefly.dump \
  firefly-db:/tmp/firefly.dump
# Or from the local pre-hook path if already on disk:
# docker cp ./backups/firefly.dump firefly-db:/tmp/firefly.dump
docker exec -i firefly-db pg_restore -U firefly -d firefly --clean --if-exists \
  /tmp/firefly.dump

# Optional: restore uploaded attachments if they matter
cp -a /tmp/restore-firefly/var/lib/docker/volumes/firefly_upload/_data/. \
  /var/lib/docker/volumes/firefly_upload/_data/

docker compose up -d

Verify: Open https://firefly.infra.realemail.app, confirm accounts and transactions. Importer: https://firefly-import.infra.realemail.app.

ARA (tier 2)

SQLite database with Ansible run history. Regeneratable but useful to keep. backup.yml selects /var/lib/docker/volumes/ara_ara-data/_data; Compose mounts ara-data at /opt/ara. The repository does not establish the SQLite filename or its relative layout. Review the actual staged data and matching application settings; never guess a database filename.

The bounded REC-002 preparation procedure is source under review in PR #473, not evidence of deployment or an application recovery test. Read the recovery-volume preparation contract before proceeding. Run only in a root Bash shell on the Linux recovery host with the full reviewed checkout at /opt/homelab, not on the editor workstation.

First select and stage an explicit reviewed snapshot using the existing path-set rules. The staging root must be a new absent absolute path; preserve prior attempts.

umask 077
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/ara
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed full snapshot ID: " SNAPSHOT_ID || exit 1
read -r -p "New absent absolute staging root: " STAGING_ROOT || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/ara/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target "$STAGING_ROOT" || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL || exit 1
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/ara/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target "$STAGING_ROOT" --restore || exit 1

Stop and review the full staged recovery set. The only copy source is $STAGING_ROOT/var/lib/docker/volumes/ara_ara-data/_data, not the staging wrapper. Keep that trusted tree and its parents stable throughout validation and preparation. Missing, malformed, symlinked or unsupported data is a stop condition, not permission to create empty directories, follow links or rewrite the backup to pass checks.

In the same shell, provide the reviewed database path relative to that _data directory and a new unused name starting with ara-recovery-. Follow the contract's path, name and stable-tree constraints. The owner must vet and separately preload a digest-pinned utility image with /bin/sh and GNU cp 9.5+ supporting --update=none-fail, with no declared VOLUME. No image or digest is supplied by this procedure.

read -r -p "Reviewed database path relative to staged _data: " ARA_DATABASE || exit 1
read -r -p "New unused ara-recovery- name: " RECOVERY_VOLUME || exit 1
read -r -p "Vetted preloaded copy image (name@sha256:digest): " COPY_IMAGE || exit 1
python3 /opt/homelab/backups/prepare-recovery-volume.py \
  --service ara --staging-root "$STAGING_ROOT" --volume "$RECOVERY_VOLUME" \
  --copy-image "$COPY_IMAGE" --ara-database "$ARA_DATABASE" || exit 1

Without --prepare, this is a filesystem-only JSON plan: no Docker calls, copy or volume creation. It verifies layout and a regular file's SQLite header, not SQLite integrity, schema compatibility or usable ARA history. A plan does not prove the target is absent or the image is loaded. Review its exact source, database path, image and destination before approving preparation only. The owner must also review the local daemon endpoint: an existing non-symlink Unix socket with non-symlink parents. Docker context/TLS overrides must be unset as described in the contract; the helper fails closed rather than choosing a different daemon.

read -r -p "Reviewed local Docker endpoint (unix:///absolute/socket): " DOCKER_HOST || exit 1
export DOCKER_HOST
read -r -p "Type PREPARE after owner approval of this exact plan: " PREPARE_APPROVAL || exit 1
test "$PREPARE_APPROVAL" = PREPARE || exit 1
python3 /opt/homelab/backups/prepare-recovery-volume.py \
  --service ara --staging-root "$STAGING_ROOT" --volume "$RECOVERY_VOLUME" \
  --copy-image "$COPY_IMAGE" --ara-database "$ARA_DATABASE" --prepare || exit 1

Preparation creates only a new labelled, unused volume, verifies ownership and emptiness, and copies through a strict read-only source bind with --pull=never and --network=none. It never stops/starts ARA, overwrites/deletes an existing volume or cuts over. Preserve any failed/partial destination; do not reuse it. After failure or interruption, check for an ongoing copy before any retry/manual action. Keep ara_ara-data unchanged for rollback.

Preparation is not recovery acceptance. Separately approve isolated SQLite integrity and application checks, including any journal/WAL files, compatible ARA version/settings, copied content/metadata fidelity, ownership, API/UI access and expected historical runs. Do not validate by starting the production service on the new volume. Before production cutover, the owner must approve the results, maintenance and rollback plan, and a reviewed GitOps change to both services/ara/compose.yml's volume mapping and services/ara/backup.yml's absolute data path. An ad-hoc Compose override can be reverted by the next deployment while backups keep reading the old volume. The preparer does not change either file or make its temporary volume a production default. Verify the resulting backup coverage after any separately approved cutover, and retain the original volume until rollback retention is approved.

Homepage (tier 2)

Config is generated from inventory — backup is convenience only.

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/homepage
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/homepage/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-homepage || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/homepage/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-homepage --restore || exit 1

Validate all staged config before any copy. Regeneration is a separate alternative, not a helper action; review its output before deploying it.

set -euo pipefail
# Instead of copying restored config, regenerate from reviewed inventory:
cd /opt/homelab || exit 1
uv run python inventory/generators/render-homepage.py

AdGuard (tier 2)

backup.yml protects config only: /var/lib/docker/volumes/adguard_adguard-conf/_data and the archived /opt/homelab/services/adguard/unbound.conf. AdGuardHome.yaml belongs in the config directory. The adguard-work volume, including query history, is not backed up. Do not clear, replace or claim to restore it from this recovery set. Current Compose does not mount unbound.conf; copying the archive back into the checkout would not restore effective Unbound configuration.

The bounded REC-003 procedure is source under review in PR #473, not evidence of deployment or an application recovery test. Read the recovery-volume preparation contract. Use a root Bash shell on the Linux recovery host, the full reviewed checkout at /opt/homelab, and a new absent absolute staging root. Preserve prior attempts.

Select a complete explicit snapshot. For legacy split snapshots, apply the existing path-set rules: repeat --snapshot for both reviewed same-host roots in both helper calls below, with no missing, extra or overlapping roots. The archived Unbound file still belongs to that staging set, even though it is not mounted or selected for volume preparation.

umask 077
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/adguard
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed complete-path-set full snapshot ID: " SNAPSHOT_ID || exit 1
read -r -p "New absent absolute staging root: " STAGING_ROOT || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/adguard/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target "$STAGING_ROOT" || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL || exit 1
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/adguard/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target "$STAGING_ROOT" --restore || exit 1

Stop and validate all staged paths and the DNS recovery set. Keep the full trusted staging tree and its parents stable throughout validation and preparation. The only copy source is $STAGING_ROOT/var/lib/docker/volumes/adguard_adguard-conf/_data; never substitute the staging wrapper, work directory or archived Unbound file. Missing, malformed, symlinked or unsupported data requires review, not automatic fixes or an empty replacement config.

In the same shell, select a new unused name starting with adguard-conf-recovery- and follow the contract's path, name and stable-tree constraints. The owner must vet and separately preload a digest-pinned image with /bin/sh and GNU cp 9.5+ supporting --update=none-fail, with no declared VOLUME. Do not substitute an unreviewed tag or invented digest.

read -r -p "New unused adguard-conf-recovery- name: " RECOVERY_VOLUME || exit 1
read -r -p "Vetted preloaded copy image (name@sha256:digest): " COPY_IMAGE || exit 1
python3 /opt/homelab/backups/prepare-recovery-volume.py \
  --service adguard --staging-root "$STAGING_ROOT" --volume "$RECOVERY_VOLUME" \
  --copy-image "$COPY_IMAGE" || exit 1

Without --prepare, the helper emits a filesystem-only JSON plan, with no Docker calls or mutation. It checks layout and a parseable YAML mapping in AdGuardHome.yaml, not AdGuard schema validity, DNS semantics or recoverability. It does not establish target availability or image preload. Review the exact source, image and destination before owner approval of preparation only. Review the local daemon endpoint too: an existing non-symlink Unix socket with non-symlink parents. Docker context/TLS overrides must be unset as described in the contract; the helper refuses them rather than choosing another daemon.

read -r -p "Reviewed local Docker endpoint (unix:///absolute/socket): " DOCKER_HOST || exit 1
export DOCKER_HOST
read -r -p "Type PREPARE after owner approval of this exact plan: " PREPARE_APPROVAL || exit 1
test "$PREPARE_APPROVAL" = PREPARE || exit 1
python3 /opt/homelab/backups/prepare-recovery-volume.py \
  --service adguard --staging-root "$STAGING_ROOT" --volume "$RECOVERY_VOLUME" \
  --copy-image "$COPY_IMAGE" --prepare || exit 1

Preparation creates only a new labelled, unused volume after target checks, verifies ownership and emptiness, and copies through a strict read-only source bind with --pull=never and --network=none. It does not stop/start AdGuard or Unbound, overwrite/delete volumes, restore work data or change production mounts. Preserve failed/partial destinations and the original adguard_adguard-conf and adguard_adguard-work volumes for rollback; do not reuse a failed destination. After failure or interruption, check for an ongoing copy before any retry/manual action.

Preparation is not recovery acceptance. Separately approve copied content and metadata fidelity checks and an isolated test of compatible AdGuard config, upstream 172.21.0.3, rewrites, filtering and DNS resolution without binding the production listener. Review the missing work-data impact and the actual Compose/image-provided Unbound configuration; an archived file is not evidence of effective settings. Before production cutover, approve test results, DNS continuity and rollback, and reconcile both the GitOps volume mapping in services/adguard/compose.yml and the absolute config-volume path in services/adguard/backup.yml. An ad-hoc override risks the next deployment reverting the mapping while backups keep reading the old volume. The preparer changes neither file and does not select a new production default.

After a separately approved cutover, verify client resolution through 192.168.6.17:53, the Authentik-protected UI, rewrites against inventory and backup coverage of the active config volume. Keep the original volumes until rollback retention is approved; this procedure authorizes no cutover or cleanup.

Monitoring (tier 2)

Grafana user customizations outside code-managed provisioning and the GitHub Actions collector SQLite state used for restart-safe event deduplication.

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/monitoring
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/monitoring/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-monitoring || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/monitoring/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-monitoring --restore || exit 1

Stop here: validate both staged volumes and their SQLite databases in isolation. Do not stop services or delete/copy data until all validation passes and the owner approves an application recovery plan. These live-copy steps are not certified here.

set -euo pipefail
# Replace Grafana data volume only after the validation and approval gate
cd /opt/homelab/services/monitoring || exit 1
docker compose stop grafana github-actions-collector

docker run --rm \
  -v monitoring_grafana-data:/data \
  -v /tmp/restore-monitoring/var/lib/docker/volumes/monitoring_grafana-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/* /data/"

docker run --rm \
  -v monitoring_collector-data:/data \
  -v /tmp/restore-monitoring/var/lib/docker/volumes/monitoring_collector-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/* /data/"

docker compose --env-file .env up -d grafana github-actions-collector

Note: Dashboards provisioned from monitoring/grafana/dashboards/ are restored automatically on Grafana startup. Only user-created dashboards and annotations need the Grafana volume restore. If no collector backup exists, start it with an empty volume; the initial lookback repopulates state but can temporarily re-observe completed jobs.

LiteLLM (tier 2)

REC-005 in PR #474 replaces hot PGDATA backup/recovery with PostgreSQL 16 logical generations for spend logs, virtual keys and key metadata. This is a source contract, not a completed deployment or application restore. The manifest now protects /var/backups/postgres-logical/litellm instead of /var/lib/docker/volumes/litellm_litellm-db-data/_data. Tier 2 remains local-only; this change adds no offsite copy. Provider usage may reconstruct some billing history, not virtual-key identities or configuration. Provider/master/database secrets still require separate private SOPS recovery and rendering. Regenerate .env from recovered SOPS using the render-only LiteLLM deployment-input refresh, not the Setup section's blanket Compose start. Preserve the existing raw POSTGRES_PASSWORD; the renderer derives POSTGRES_PASSWORD_URLENCODED for the proxy URL while the database receives the raw credential. Do not store the derived key in SOPS, reuse an old .env without it, or rotate the password to pass validation. Before live cutover, privately verify exact credential byte/string equivalence through Compose and the decoded URL, and verify database authentication. This representation change does not authorize ALTER ROLE or a password rotation.

The shared producer defaults to an offline plan with --service litellm; the scheduled pre-hook explicitly invokes python3 with --backup and DOCKER_HOST=unix:///run/docker.sock. Each completed private, immutable generation under /var/backups/postgres-logical/litellm, outside the shared checkout, contains litellm.dump, litellm.dump.sha256 and manifest.json, binding service, database/user, PostgreSQL server/client major 16, capture time, filename, size and SHA-256; a valid supplied systemd INVOCATION_ID is also bound as invocation_id. Exports require root and pre-provisioned root:root mode-0700 namespace/service directories. Backup-client's logical-postgres.yml provisions them before units and refuses foreign-owned namespaces and symlinks; the producer never changes existing directory permissions. The producer, pruner and verifier require Python 3.11+; the producer and pruner also need host PyYAML. Normal role apply must ensure Python 3/PyYAML and pass a read-only version/import check before units; check mode cannot certify missing dependencies. Require full backup convergence with mounts, repositories and credentials ready. A systemd-only refresh is valid only after those base prerequisites, not as bootstrap. Read-only host metadata observations are not database-query, export or restore evidence.

The bounded-storage follow-up implements bounded storage and scheduled retention, not the initial PR's unbounded local growth or absent headroom enforcement. Owner policy review is required before merge and scheduling: merge/convergence enables future scheduled local deletion. No live operation is performed here.

LiteLLM keeps the seven newest validated completed generations, not seven days; at 14, exports stop without deleting data. Dumps are capped at 512 MiB, catalog scratch at 32 MiB, each metadata file at 64 KiB. Preflight budgets the maximum dump + catalog + 2 × 64 KiB metadata and 5 inodes. The same max(ceil(15% of total bytes), 8 GiB) + 16 GiB byte floor/reserve and max(ceil(5% of total inodes), 100,000) inode floor apply plus prospective allocation, before export and during writes. Invalid accounting and failed .staging-*/.deleting-*, unknown/foreign/corrupt/oversize entries or unsafe links block new exports and pruning; they are never automatically deleted or repaired. The lock coordinates cooperating producers/pruners, not unrelated writers or quotas. Do not remove evidence or override checks to force a capacity pass.

The dedicated pruner's --service litellm default is a read-only filesystem plan, not a cleanup recipe. Explicit --prune is only the direct systemd ExecStartPost after successful NAS backup/forget (Paperless additionally requires B2 copy/forget), with matching invocation, current ControlPID, activating/start-post, and exactly one invocation-bound newest generation. Failed required prior stages prevent it. Restic retention and historical host,paths groups are unchanged; no automatic legacy-data or historical-snapshot cleanup is added. See the logical backup contract. For the new two-job critical/page PostgresLogicalBackupRunFailed alert, follow the failure triage. The existing page route is source configuration, not live delivery acceptance; OPS-042 remains open.

Stage a reviewed recovery set

Use the prerequisites above, Python 3.11+ and a full reviewed checkout, not standalone copied scripts. Recover restic access separately; staging/verification does not restore application secrets or render .env. Latest snapshots can contain 14 generations before pruning, up to 7 GiB of LiteLLM dump payload plus metadata/filesystem overhead. Budget the full staging copy alongside live data and verifier memory/tmpfs; the two-service live dump ceiling of 7.875 GiB is not a total recovery-space bound.

Historical roots differ: older snapshots contain raw PGDATA, not these logical generations. Use their separately reviewed historical manifest with guarded restore-snapshot.py staging, and explicit same-host disjoint IDs for any split set. Do not weaken the new manifest to accept the old roots. Historical hot raw files are unproven physical backups: this repair neither validates them nor invents a logical dump for them. If no trusted logical generation exists, stop for a separate recovery assessment; preserve the files and snapshots as evidence.

The following commands are for a reviewed snapshot matching the new manifest:

umask 077
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/litellm
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed full snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/litellm/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-litellm || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/litellm/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-litellm --restore || exit 1

Validate all staging, then select one completed generation explicitly. Keep its dump, sidecar and manifest together, private, stable and free of symlink traversal; never substitute a local checkout dump or an implicit newest file.

Plan and run an isolated database smoke test

The verifier validates the selected staged file. Its default is filesystem-only: no Docker calls, pulls, imports or mutation. Only custom-format PGDMP archives are accepted. Choose an owner-vetted PostgreSQL 16 image with an exact name@sha256:digest reference and separately preload it on the intended trusted local daemon before explicit verification. It must support the verifier's postgres user, standard entrypoint and PG16 layout; see the linked contract for the image checks.

read -r -p "Absolute staged generation path ending in /litellm.dump: " LITELLM_DUMP
case "$LITELLM_DUMP" in
  /tmp/restore-litellm/var/backups/postgres-logical/litellm/*/litellm.dump) ;;
  *) echo "Stop: select a generation from the reviewed staging target" >&2; exit 1 ;;
esac
read -r -p "Vetted preloaded PostgreSQL 16 image (name@sha256:digest): " POSTGRES_IMAGE
python3 /opt/homelab/backups/verify-postgres-backup.py \
  --service litellm --dump "$LITELLM_DUMP" --image "$POSTGRES_IMAGE" || exit 1

After plan review, require owner approval of the daemon and memory/tmpfs capacity. Set an explicit reviewed DOCKER_HOST=unix:///absolute/socket with safe ownership and no symlink traversal; review and unset conflicting Docker context/TLS overrides. Follow the full endpoint/image requirements in the linked logical backup contract, then explicitly request verification:

read -r -p "Type VERIFY to approve a disposable database smoke test: " VERIFY_APPROVAL
test "$VERIFY_APPROVAL" = VERIFY || exit 1
python3 /opt/homelab/backups/verify-postgres-backup.py \
  --service litellm --dump "$LITELLM_DUMP" --image "$POSTGRES_IMAGE" --verify || exit 1

This creates a fresh owned PostgreSQL 16 container with fixed 1 GiB data tmpfs / 2 GiB memory, --network=none, no ports, no host data mounts and no application writers. Even dumps within the producer limit can expand beyond these bounds; the fixed 1 GiB smoke is not capacity certification, and there is no sizing CLI override. Archive checks, pg_restore --exit-on-error --single-transaction --no-owner --no-acl and a positive ordinary application-table count are a database smoke test, not full LiteLLM recovery; expected rows and original grants are not checked. Cleanup removes only the exact owned temporary container, never backups, staged files or live volumes. No automatic verifier wiring is added; a generic weekly file count is not application-restore certification, and LiteLLM remains outside the tier-1 weekly rotation.

Separate production cutover gate

An owner-approved cutover must use a fresh persistent database with compatible PostgreSQL/application versions, separately recovered/rendered secrets, reviewed Compose/storage/backup mappings and a rollback plan preserving the old data. The verifier's disposable database is never promoted. Keep the proxy, background jobs and all other application writers off until successful import and checks; account for restart/deployment automation. Do not clear live PGDATA, blindly drop a database, or start the full Compose stack before import. The render-only secret refresh above is a prerequisite, not permission to run Setup's blanket start or reopen writers; an old .env missing the generated URL password fails closed.

Require isolated application checks of expected virtual-key metadata/authorization, spend/budget history, /v1/models, Admin UI and metrics before approved reopening. A nonempty database or HTTP 200 alone is insufficient. Do not expose key values in evidence. Restic access, secret recovery/rendering, database smoke, application validation and production acceptance remain distinct gates. Real PostgreSQL 16 export/import, isolated application acceptance and rollback rehearsal remain open. LiteLLM gains no offsite coverage; required Paperless offsite recovery remains separate.

Stirling-PDF (tier 2)

Restores configs, OCR tessdata, and pipeline volumes. Uploaded PDFs are not retained by design.

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/stirling-pdf
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/stirling-pdf/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-stirling || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/stirling-pdf/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-stirling --restore || exit 1

Stop here: validate all staged volumes before any stop/delete/copy. The following application recovery context needs separate owner approval and is not certified here.

set -euo pipefail
cd /opt/homelab/services/stirling-pdf || exit 1
docker compose down

for vol in configs tessdata pipeline; do
  docker run --rm \
    -v "stirling-pdf_${vol}:/data" \
    -v "/tmp/restore-stirling/var/lib/docker/volumes/stirling-pdf_${vol}/_data:/backup:ro" \
    alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"
done

docker compose up -d

Verify: Open https://stirling.infra.realemail.app (Authentik) and confirm saved pipelines/settings.

Phoenix (tier 2)

Phoenix runs on prox LXC 124, not infra-services. Restore on that host.

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/phoenix
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/phoenix/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-phoenix || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/phoenix/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-phoenix --restore || exit 1

Stop here: validate every staged root and database consistency in isolation. These raw-volume/application steps are not certified by the path-set repair; obtain owner approval before any live stop/delete/copy.

set -euo pipefail
cd /opt/homelab/services/phoenix || exit 1
docker compose down

docker run --rm \
  -v phoenix_phoenix-db-data:/data \
  -v /tmp/restore-phoenix/var/lib/docker/volumes/phoenix_phoenix-db-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"

docker run --rm \
  -v phoenix_phoenix-data:/data \
  -v /tmp/restore-phoenix/var/lib/docker/volumes/phoenix_phoenix-data/_data:/backup:ro \
  alpine sh -c "rm -rf /data/* && cp -a /backup/. /data/"

docker compose up -d

Verify: https://phoenix.infra.realemail.app loads through Authentik and LiteLLM's full trace test can query the litellm project.

Wazuh (retired)

The stack and its Docker volumes were removed. Do not restore it. Historical restic snapshots remain in the existing repositories and were not forgotten.

Order Ledger (tier 1)

Backs up /var/lib/homelab/orders. The pre-hook writes a consistent SQLite copy to orders.backup.sqlite3. Live orders.sqlite3 files and gmail_token.json are excluded. The Google OAuth client is SOPS state under /etc/homelab/orders, not this restic path.

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/orders
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/orders/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-orders || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/orders/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-orders --restore || exit 1

Stop here: validate all staging, including the SQLite backup, before any stop or live copy. The following application recovery context needs separate owner review; this path-set repair does not certify database replacement or token recovery.

set -euo pipefail
cd /opt/homelab/services/orders || exit 1
docker compose --env-file compose.env stop

sudo install -d -o 10001 -g 10001 -m 0750 /var/lib/homelab/orders
sudo cp -a /tmp/restore-orders/var/lib/homelab/orders/orders.backup.sqlite3 \
  /var/lib/homelab/orders/orders.sqlite3
sudo chown 10001:10001 /var/lib/homelab/orders/orders.sqlite3

sudo env ANSIBLE_CONFIG=/opt/homelab/infra/ansible/ansible.cfg \
  ansible-playbook -i /opt/homelab/infra/ansible/inventory/generated.yml \
  /opt/homelab/infra/ansible/playbooks/site.yml --limit infra-services \
  --tags orders,orders-secrets \
  -e ansible_connection=local
docker compose --env-file compose.env up -d

Sign in through Authentik and use Connect Gmail again. The refresh token is not in the backup. Confirm orders-bot becomes healthy after the next Telegram poll.

Authentik outpost (tier 3)

The infra-services outpost is stateless. Its token is SOPS-managed, and the authoritative provider/outpost state lives in the Authentik server on saltierpoop.

set -euo pipefail
cd /opt/homelab/services/authentik-outpost || exit 1
python3 /opt/homelab/scripts/render-sops-env.py \
  --source .env.sops.yaml --output .env --require AUTHENTIK_TOKEN || exit 1
docker compose up -d

If the token is invalid or lost, recreate it from the Authentik admin flow in authentik-infra-admin-setup.md, update .env.sops.yaml, and redeploy the outpost.

External Services (not on infra-services)

Saltbox Secrets (saltierpoop)

The encrypted sources are secrets/appliances/saltierpoop/accounts.sops.yaml and settings.sops.yaml. The destinations are /srv/git/saltbox/accounts.yml and /srv/git/saltbox/settings.yml, as selected by saltierpoop.yml. Re-apply the saltbox-host secrets tasks after recovering the host's age key. The obsolete roles/saltbox-host/files/accounts.sops.yaml and /opt/saltbox/ paths are not used. Homelab does not restore or redeploy Saltbox containers.

Authentik DB (saltierpoop)

The external-authentik-db snapshot contains a gzip-compressed pg_dumpall from the someone PostgreSQL role. Its captured path is /var/backups/external-staging/authentik-db/authentik.sql.gz, not the remote host's temporary dump path.

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/external-authentik-db
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/authentik-db.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-authentik || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/authentik-db.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-authentik --restore || exit 1
gzip -t /tmp/restore-authentik/var/backups/external-staging/authentik-db/authentik.sql.gz

Validate every staged root and that dump against an isolated compatible PostgreSQL instance first. Do not transfer data, stop writers, or import into production before all staging validates and the owner approves a separate application recovery plan. The following import context is not certified by the path-set repair. For a live recovery, transfer the reviewed dump to saltierpoop, stop Authentik application writers through Saltbox, and restore to the intended database cluster. A cluster-wide SQL dump includes roles and databases; review conflicts before applying it to a populated cluster. Do not feed compressed bytes to psql. The import form, once the recovery target is prepared, is:

read -r -p "Absolute path to the validated Authentik dump on this host: " AUTHENTIK_DUMP
gzip -dc "$AUTHENTIK_DUMP" \
  | docker exec -i authentik-postgres psql -X -v ON_ERROR_STOP=1 -U someone -d postgres

Restart Authentik through Saltbox and verify login, OIDC and outpost health. This audit checked the fetch script and manifest, not a live Authentik restore.

PostgreSQL Companion (LXC 102)

The external-postgresql-companion repo contains custom-format dumps for companion, tdler, and reddit_explorer, cluster globals, expected application-table row counts, and SHA256SUMS. Its primary encrypted repo is on the Whrrr NAS and, as tier 1, is also copied to B2.

# Restore from the primary NAS repository.
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/external-postgresql-companion
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/postgresql-companion.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-postgresql-companion || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/postgresql-companion.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-postgresql-companion --restore || exit 1

# Validate without opening ports or contacting the live PostgreSQL LXC.
sudo /opt/homelab/backups/restore-drills/postgresql-companion.sh \
  /tmp/restore-postgresql-companion/var/backups/external-staging/postgresql-companion

For total-loss recovery, use the offsite pattern with repository suffix external-postgresql-companion and absolute manifest /opt/homelab/backups/external/postgresql-companion.yaml; review IDs in B2, then stage into an absent /tmp/restore-postgresql-companion target. Do not reuse a local attempt. The weekly restore-test includes this isolated PostgreSQL 18 container drill; the new helper wiring is source work, not evidence of a deployed or successful run. Only after all staging and the drill validate should an operator seek approval for a live-cluster restore. The drill removes its temporary container and does not contact 192.168.6.252; the staging helper never cleans up recovery evidence.

HAOS Config

Full snapshots (~2 GiB). NAS (/mnt/infra-backups/homelab-restic) is the primary restic repo; B2 is a safety net only (keep-last 2 on B2; weekly full + daily incrementals). Local staging under /var/backups/external-staging is cleared after a successful backup.

From NAS (primary):

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/external-haos-config
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/haos-config.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-haos || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/haos-config.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-haos --restore || exit 1

From B2 (total-loss / offsite DR):

source /etc/restic/b2.env
read -r -p "Recovery B2 bucket name: " B2_BUCKET
export RESTIC_REPOSITORY="b2:${B2_BUCKET}:external-haos-config"
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed offsite snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/haos-config.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-haos || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/haos-config.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-haos --restore || exit 1

NAS and B2 are alternatives: /tmp/restore-haos must be absent for either attempt. Validate all staged files and the HAOS archive before any upload or application restore; root existence does not certify the archive. After separate owner approval, restore the snapshot .tar via the HA Supervisor API or manually upload through the HAOS UI at Settings > System > Backups. This application step is not certified by the path-set repair.