Backup alert triage

Use this runbook for restic backup and restore-test alerts. The first goal is to decide which layer is failing: metrics collection, systemd scheduling, fetch hooks, NAS restic snapshots, B2 copy targets, or restore validation.

Do not delete, move, or prune backup artifacts while triaging without explicit owner approval.

Entry points

Surface	Use
Backup / Restic Triage	Backup health overview and per-service drilldown
Alertmanager	Active backup alert groups and silences
Restore procedures	Per-service restore commands
DR from Zero	Full rebuild order

Scope

The backup-client role currently runs on Ansible docker_hosts except unprivileged Proxmox LXCs (proxmox_guests): today that means infra-services only. Phoenix and sonarqube stay in docker_hosts for other roles, but they cannot mount Whrrr /volume6/infra-backups from inside the CT, so in-guest restic is disabled until a host-side or idmap-safe path exists. Those guests rely on Proxmox vzdump / manual archive in the meantime.

ResticMetricsMissing therefore scopes to instance="infra-services" only (not phoenix/sonarqube). Backup manifests may still list phoenix and sonarqube for documentation; they are not applied by ansible-pull on those hosts.

Expected backup jobs from current manifests:

• Tier 1 on infra-services: external-authentik-db, external-saltbox-secrets, komodo, oneuptime, oneuptime-incident-adapter, paperless, and traefik.
• Tier 2 on infra-services: adguard, ara, external-haos-config, homepage, litellm, monitoring, paperless-stirling-bridge, stirling-pdf, and wazuh.
• Tier 2 appliance-local jobs (manifest only until LXC NFS is fixed): phoenix/phoenix and sonarqube/sonarqube.

harbor-registry is documented in backups/external/harbor-registry.yaml, but it is not automatically discovered by backup-client today because it has no fetch_script and the host is not in docker_hosts.

The weekly restore-test timer runs only on infra-services. Each execution selects the next local tier-1 service manifest in sorted rotation: komodo, oneuptime, oneuptime-incident-adapter, paperless, or traefik. State lives in /var/lib/restic-restore-test/next-index and advances even when a drill fails so one bad repo cannot starve the others. With five candidates and a weekly cadence, each repo is exercised about every 35 days — matching the RestoreTestStale window. External tier-1 jobs and all tier-2 jobs remain outside this drill.

Metrics

The collector at /usr/local/bin/restic-metrics writes /var/lib/node_exporter/textfile_collector/restic.prom. node_exporter reports the textfile mtime with a file label that may be the full path, not just restic.prom.

Metric	Meaning
restic_backup_expected	Job expected from a backup manifest
restic_backup_copy_expected	Expected copy target, currently b2 or nas
restic_backup_last_success_timestamp_seconds	Last snapshot timestamp in the local repo
restic_backup_snapshot_count	Snapshot count in the local repo
restic_backup_last_size_bytes	Size of the latest local snapshot
restic_backup_last_run_result	Last systemd result label for the backup unit
restic_backup_last_run_exit_code	Last systemd ExecMainStatus
restic_backup_last_run_timestamp_seconds	Last backup unit completion time
restic_backup_last_run_duration_seconds	Last backup unit duration
restic_backup_timer_active	Whether the systemd timer is active
restic_backup_timer_next_run_timestamp_seconds	Next timer fire time
restic_restore_test_last_success_timestamp_seconds	Last successful restore drill
restic_restore_test_success	Whether the last restore drill passed

Alert response

ResticMetricsMissing

Prometheus can scrape node_exporter, but restic.prom is absent.

Check:

systemctl status restic-metrics.timer restic-metrics.service
ls -l /var/lib/node_exporter/textfile_collector/restic.prom
journalctl -u restic-metrics.service --since "2 hours ago"

If the file exists locally but the alert still fires, check Prometheus for the raw textfile label:

node_textfile_mtime_seconds{file=~"(.*/)?restic\\.prom"}

Likely causes: backup-client did not converge, node_exporter textfile path is missing, or the metrics script exits before writing the file.

ResticMetricsStale

restic.prom exists but has not refreshed in over an hour.

Check:

systemctl list-timers 'restic-metrics*'
systemctl status restic-metrics.timer
journalctl -u restic-metrics.service --since "2 hours ago"
/usr/local/bin/restic-metrics

If a manual run fails, fix that error before restarting the timer.

ResticExpectedBackupMissing

A job is declared by a manifest, but no snapshot-count metric exists for that service.

Check:

systemctl status restic-backup-<service>.timer restic-backup-<service>.service
ls -la /mnt/infra-backups/homelab-restic/<service>
RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Likely causes: repo initialization failed, the service path does not exist on that host, or restic-metrics cannot inspect the repo.

ResticBackupRunFailed

The last systemd run finished with a non-success result.

Check the unit and journal:

systemctl status restic-backup-<service>.service
journalctl -u restic-backup-<service>.service --since "24 hours ago"

For external jobs, also check the fetch hook under backups/external/. Failures can come from SSH, SOPS decrypt, Home Assistant token/API behavior, B2 copy, NFS mount issues, or restic prune.

Stale locks (repository is already locked, exit 11)

If the journal shows repository is already locked by PID <n> and unable to create lock in backend (restic exit 11), an earlier restic op (interrupted backup/copy or a manual run) left a lock behind. The restic backup step still succeeds (non-exclusive lock), but forget --prune needs an exclusive lock and fails, marking the whole unit failed.

The backup units now run restic unlock before each prune (NAS primary, and before the B2 copy) to clear locks whose process is dead, so this should self-heal on the next run. One interrupted op can leave a lock on both the NAS and B2 repos — clear both if triaging by hand:

# NAS primary repo
sudo RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password restic unlock
# B2 repo (source b2.env for credentials)
sudo bash -c '. /etc/restic/b2.env && \
  RESTIC_REPOSITORY=b2:<bucket>:<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password restic unlock'
sudo systemctl start restic-backup-<service>.service

ResticBackupTimerInactive

The service has an expected backup job, but the timer is not active.

Check:

systemctl list-timers 'restic-backup-*'
systemctl enable --now restic-backup-<service>.timer

If the timer file is missing, rerun Ansible for the backup-client role instead of creating an unmanaged timer.

ResticBackupStale

The latest local snapshot is older than 36 hours. This alert is scoped to jobs that emit restic_backup_expected on the same host; stray or retired repo directories can still appear in raw metrics but should not alert unless the job is expected there.

ResticBackupStale and ResticBackupFailed share one OneUptime incident per canonical instance,backup_service. The critical empty-repository threshold inhibits only the warning for that same job; unrelated backup incidents remain visible. The adapter accepts only host/job pairs declared by current backup manifests; synthetic acceptance uses the reserved adapter-acceptance-* job namespace.

Check:

RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots
systemctl list-timers restic-backup-<service>.timer
journalctl -u restic-backup-<service>.service --since "48 hours ago"

If the last run succeeded but the snapshot age is still stale, verify the backup paths exist and restic is not backing up an empty or wrong path.

ResticBackupFailed

The local repo has zero snapshots. Treat this as critical for tier 1 and urgent for tier 2. Like stale backups, this alert is scoped to expected jobs on that host.

Check repo initialization and run one controlled backup after fixing path or credential issues:

RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots
systemctl start restic-backup-<service>.service
journalctl -u restic-backup-<service>.service -n 200 --no-pager

Restore-test alerts

RestoreTestMetricsMissing means the canonical host has never emitted or has lost its restore-test metric. RestoreTestStale means the weekly drill has not succeeded recently, and RestoreTestFailed means the latest drill failed.

Both thresholds share one host-level OneUptime incident keyed by canonical instance. Restore drills rotate one tier-1 repository per run, so the last selected repository is diagnostic context (restore_service), not durable incident identity. The critical failure inhibits only the overdue warning on the same host.

Check:

systemctl status restic-restore-test.timer restic-restore-test.service
journalctl -u restic-restore-test.service --since "45 days ago"
/opt/homelab/backups/restore-test.sh

If a restore failed for a specific service, continue with that service's section in restore procedures.

Copy targets

Primary restic repos live on Whrrr NFS (/mnt/infra-backups/homelab-restic). Tier-1 (and explicit offsite: true) jobs also copy to B2. A failed B2 copy should make the systemd backup unit non-success because copy operations run as ExecStartPost.

Check B2:

. /etc/restic/b2.env
RESTIC_REPOSITORY=b2:<bucket>:<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Check NAS:

mountpoint /mnt/infra-backups
RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Validation

After remediation:

1. Run /usr/local/bin/restic-metrics.
2. Confirm restic.prom mtime updates.
3. Confirm the Backup / Restic Triage dashboard shows the expected service.
4. Confirm Prometheus alert state resolves.
5. For real data-loss risk, run a scoped restore to /tmp/restore-<service> and follow restore procedures.