Public edge incident response (Cloudflare + Traefik + Authentik)¶

Runbook for SEC-009: credential leak, certificate failure, or database corruption on the public-facing chain that serves *.realemail.app and *.infra.realemail.app.

Related: Restore (per-service restic), DR from Zero, Secrets compromise, Traefik README.

Dependency chain¶

flowchart LR
    CF[Cloudflare DNS + proxy]
    TR[Traefik infra-services + saltierpoop]
    AK[Authentik on saltierpoop]
    APPS[Protected apps behind Traefik]

    CF -->|DNS-01 ACME| TR
    CF -->|orange-cloud optional| TR
    TR -->|forward auth| AK
    AK --> APPS

Layer	What breaks if it fails	Typical symptom
Cloudflare API token	Traefik cannot renew Let's Encrypt certs via DNS-01	TLS errors, cert expiry alerts
Traefik `acme.json`	Same — lost or corrupt ACME account state	Sudden TLS failures after restart
Authentik DB	SSO / forward-auth broken	401/502 on protected routes, login loops
Traefik config	Routing wrong or down	404/502 for all hostnames

Tier-1 backups cover Traefik state, Komodo, and Authentik DB (external fetch from saltierpoop). Cloudflare account credentials are not in restic — they live in SOPS, GitHub Actions secrets, and 1Password.

Scenario A — Cloudflare API token leak or rotation¶

When to use¶

Token committed to git, pasted in a ticket, or Cloudflare audit shows misuse.
Planned rotation (good hygiene every 6–12 months).
Traefik logs: Cloudflare API errors during cert renewal.

Where tokens live today¶

Location	Key name	Scope needed
`services/traefik/.env.sops.yaml`	`CF_DNS_API_TOKEN`	Zone DNS Edit for `realemail.app`
GitHub Actions `docs.yml`	`CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`	Pages deploy (separate token recommended)
Saltbox inventory on saltierpoop	Per-app DNS roles	Optional — many apps use Cloudflare DNS records

Use separate tokens per consumer when possible (Traefik DNS-01 vs CI vs Saltbox).

Response steps¶

Cloudflare dashboard → My Profile → API Tokens → revoke the compromised token.
Create replacement with minimum scope:
Traefik: Zone → DNS → Edit on realemail.app (and any other zones Traefik manages).
Docs CI: Account → Cloudflare Pages → Edit only.
Update secrets (never commit plaintext):

# On dev machine with age key / sops
sops services/traefik/.env.sops.yaml
# Edit CF_DNS_API_TOKEN, save encrypted
git add services/traefik/.env.sops.yaml && git commit && git push

Redeploy Traefik on infra-services, only after separate rotation/deploy approval and the rendering prerequisites. Use authorized noninteractive sudo for the root-only automation key and resulting root-owned 0600 .env; do not weaken key permissions. Failed decryption or conversion preserves .env and must prevent deployment. Source tests do not prove live access, target Compose compatibility or successful token rotation. Connect separately with ssh infra-services; run the complete block in the target Bash shell, not after an SSH command that may return to the local shell:

set -euo pipefail
cd /opt/homelab/services/traefik || exit 1
sudo -n python3 /opt/homelab/scripts/render-sops-env.py \
  --source .env.sops.yaml --output .env \
  --rename CF_DNS_API_TOKEN=CLOUDFLARE_DNS_API_TOKEN \
  --require CLOUDFLARE_DNS_API_TOKEN || exit 1
sudo -n docker compose up -d

Saltbox Traefik (if token shared): update inventory / accounts.yml as applicable, then sb install traefik on saltierpoop.
GitHub: Settings → Secrets → update CLOUDFLARE_API_TOKEN if CI token was rotated.
Verify renewal (force if needed):

docker logs traefik 2>&1 | tail -50 | grep -i acme
curl -sI https://traefik.infra.realemail.app | head -5

Post-incident: Cloudflare Audit Logs → confirm old token shows no further API calls.

If certs already expired before rotation¶

Traefik will request new certs once the token is valid. Browsers may need a hard refresh. If acme.json is corrupt, restore Traefik backup first (Scenario B) then rotate token.

Scenario B — Traefik `acme.json` loss or corruption¶

Current Compose stores this file at /letsencrypt/acme.json in the traefik_traefik-certs named volume. See the restore procedure.

When to use¶

acme.json deleted, zeroed, or permissions wrong (acme.json must be 600).
Traefik fails to start with ACME / lock errors.

Recovery¶

Use the shared staging prerequisites: a root Bash shell on the Linux recovery host, Python 3 with PyYAML (python3-yaml), restic on PATH, and a reviewed checkout at /opt/homelab. Both repository/password-file variables must be exported. Each staging target must be absent with existing, non-symlink parents; do not pre-create it or delete an earlier attempt to reuse it. These prerequisites also apply to Scenario C.

The producer repair in PR #469 uses one invocation for all manifest roots and tag homelab-path-set-v1, not an application-consistency guarantee. Old snapshots remain split. This is source work, not live deployment or drill evidence; the dated tabletop record below is unchanged.

Plan and stage from tier-1 restic (preferred):

# Root Bash on infra-services.
umask 077
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/traefik
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/traefik/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-traefik || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/traefik/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-traefik --restore || exit 1

The default produces metadata/preflight JSON only. For legacy split paths, repeat explicit IDs only for reviewed, disjoint same-host snapshots exactly covering the manifest. Assess their time consistency separately. --latest is scheduled verification only and refuses an incomplete newest snapshot, with no fallback or automatic mixing.

Stop here: validate all config and certificate staging before any service stop, deletion or live copy. Path completeness does not prove a valid backup. The application steps below require separate owner approval and are not certified by this path-set repair; remaining application recovery defects are not fixed here. The helper performs no cutover, promotion or cleanup. Failed staging is preserved; choose a new absent target for a retry.

set -euo pipefail
cd /opt/homelab/services/traefik || exit 1
docker compose stop traefik
docker volume create traefik_traefik-certs
docker run --rm -v traefik_traefik-certs:/certs \
  -v /tmp/restore-traefik/var/lib/docker/volumes/traefik_traefik-certs/_data:/backup:ro \
  alpine:3.22 sh -c 'cp -a /backup/. /certs/ && chmod 600 /certs/acme.json'
docker compose up -d traefik

If no snapshot: valid CF_DNS_API_TOKEN + empty acme.json lets Traefik re-issue all certs (may hit Let's Encrypt rate limits — stagger or use staging first).
Verify HTTPS on traefik.infra.realemail.app and one *.realemail.app hostname.

Scenario C — Authentik PostgreSQL corruption or loss¶

When to use¶

authentik container crash loops; Postgres errors on startup.
Accidental docker volume rm or failed upgrade.
Need point-in-time recovery after bad config change.

Backups available¶

Source	Path / job	Schedule
Tier-1 external fetch	`backups/external/authentik-db.yaml` → `fetch-authentik-db.sh`	Daily ~02:08 PDT
Restic local + B2	`restic-backup-external-authentik-db.timer`	After fetch

Staging on infra-services: /var/backups/external-staging/authentik-db/.

Recovery on saltierpoop¶

Stage a reviewed dump on infra-services before stopping writers. Use the root-shell/tool prerequisites from Scenario B. /tmp/restore-authentik must be absent; the helper creates it privately (mode 0700).

umask 077
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/external-authentik-db
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/authentik-db.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-authentik || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/backups/external/authentik-db.yaml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-authentik --restore || exit 1
gzip -t /tmp/restore-authentik/var/backups/external-staging/authentik-db/authentik.sql.gz

For B2, use the offsite pattern with this same external manifest and repository suffix external-authentik-db. Preserve failed staging and use a new absent target for retries. A fresh export from a corrupt source is not a substitute for a reviewed recovery snapshot.

Validate all staging and an isolated compatible empty-cluster restore. Gzip integrity and root existence do not prove that Authentik is recoverable. Do not stop services, delete data or copy/transfer the dump until validation passes and the owner approves a separate application recovery plan. Remaining application-specific recovery defects are not certified by this repair. After that gate, the retained transfer and stop context is:

# On infra-services, after all validation and owner approval.
scp /tmp/restore-authentik/var/backups/external-staging/authentik-db/authentik.sql.gz \
  someone@192.168.6.243:/tmp/

# On saltierpoop; review the target and Saltbox lifecycle before proceeding.
ssh someone@192.168.6.243
docker stop authentik authentik-worker 2>/dev/null || true

Restore into the separately reviewed Postgres target:

gunzip -c /tmp/authentik.sql.gz | docker exec -i authentik-postgres \
  psql -X -v ON_ERROR_STOP=1 -U someone -d postgres

This is a cluster-wide pg_dumpall export. Restore into an isolated empty compatible cluster first; roles and database creation statements can conflict with an existing target. Plan the cutover after validation rather than running a generic drop/create command against production.

Start Authentik:

docker start authentik authentik-worker
docker logs authentik --tail 30

Verify:
Login at https://auth.realemail.app (or your Authentik hostname).
Open a Traefik-protected app — forward-auth should succeed.
Check Authentik admin → Applications / Providers still present.

See restore.md § Authentik DB.

If backup is stale or missing¶

Re-bootstrap Authentik from Saltbox (sb install authentik) and re-create providers/applications manually — last resort.
Export provider config periodically (Authentik blueprints) as future hardening.

Scenario D — Full edge down (DNS + TLS + auth)¶

Work top-down:

DNS resolves? dig +short grafana.infra.realemail.app → 192.168.6.17 (LAN) or Cloudflare proxy IP (WAN).
TLS? curl -vI https://… — cert name, expiry, issuer.
Traefik routing? Traefik dashboard / access logs.
Authentik? Direct hit to auth URL bypassing apps.
App container? Komodo stack status.

Use DR from Zero only for total loss; partial incidents use sections above.

Validation log¶

Date	Check	Result
2026-06-21	`restic-backup-traefik.timer`	active
2026-06-21	`restic-backup-external-authentik-db.timer`	active
2026-06-21	Authentik staging dump	`authentik.sql.gz` ~23 MB
2026-06-21	Runbook scenarios A–C	documented
2026-06-21	Tabletop (owner)	proven — token rotation, Traefik redeploy, backup restore paths exercised

Prevention checklist¶

[x] Traefik + Authentik tier-1 timers green on infra-services
[x] Cloudflare tokens scoped per service; rotation date in 1Password
[x] acme.json backed up daily (Traefik restic timer)
[x] Authentik admin 2FA enabled
[x] This runbook exercised once (tabletop): token revoke + restore Authentik from restic

Closure¶

SEC-009 closed 2026-06-21 — see security-register.md.

Validation scope (2026-10-24)¶

The historical tabletop record above remains dated evidence. The October audit corrected certificate-volume paths and the Authentik import format against current Compose and backup sources; it did not rotate tokens or restore production data. Use the external Authentik restore procedure for the current staging path and isolated validation before a live import. The first path-set repair changes selection/staging guidance only; no new live recovery or application-consistency certification is claimed. Restic's default host,paths retention grouping leaves legacy split groups separate. No automatic deletion or grouping change is authorized: check storage, verify new backups, and obtain separate approval before cleanup.

Public edge incident response (Cloudflare + Traefik + Authentik)¶

Dependency chain¶

Scenario A — Cloudflare API token leak or rotation¶

When to use¶

Where tokens live today¶

Response steps¶

If certs already expired before rotation¶

Scenario B — Traefik acme.json loss or corruption¶

When to use¶

Recovery¶

Scenario C — Authentik PostgreSQL corruption or loss¶

When to use¶

Backups available¶

Recovery on saltierpoop¶

If backup is stale or missing¶

Scenario D — Full edge down (DNS + TLS + auth)¶

Validation log¶

Prevention checklist¶

Closure¶

Validation scope (2026-10-24)¶

Scenario B — Traefik `acme.json` loss or corruption¶