Backup alert triage

Use this runbook for restic backup and restore-test alerts. The first goal is to decide which layer is failing: metrics collection, systemd scheduling, fetch hooks, local restic snapshots, copy targets, or restore validation.

Do not delete, move, or prune backup artifacts while triaging without explicit owner approval.

Entry points

Surface	Use
Backup / Restic Triage	Backup health overview and per-service drilldown
Alertmanager	Active backup alert groups and silences
Restore procedures	Per-service restore commands
DR from Zero	Full rebuild order

Scope

The backup-client role currently runs on Ansible docker_hosts, which are infra-services and phoenix. It discovers services/*/backup.yml plus external backup definitions with fetch_script under backups/external/.

Expected backup jobs from current manifests:

• Tier 1: komodo, traefik, external-authentik-db, external-saltbox-secrets.
• Tier 2: adguard, ara, homepage, litellm, monitoring, phoenix, wazuh, external-haos-config.

harbor-registry is documented in backups/external/harbor-registry.yaml, but it is not automatically discovered by backup-client today because it has no fetch_script and the host is not in docker_hosts.

Metrics

The collector at /usr/local/bin/restic-metrics writes /var/lib/node_exporter/textfile_collector/restic.prom.

Metric	Meaning
restic_backup_expected	Job expected from a backup manifest
restic_backup_copy_expected	Expected copy target, currently b2 or nas
restic_backup_last_success_timestamp_seconds	Last snapshot timestamp in the local repo
restic_backup_snapshot_count	Snapshot count in the local repo
restic_backup_last_size_bytes	Size of the latest local snapshot
restic_backup_last_run_result	Last systemd result label for the backup unit
restic_backup_last_run_exit_code	Last systemd ExecMainStatus
restic_backup_last_run_timestamp_seconds	Last backup unit completion time
restic_backup_last_run_duration_seconds	Last backup unit duration
restic_backup_timer_active	Whether the systemd timer is active
restic_backup_timer_next_run_timestamp_seconds	Next timer fire time
restic_restore_test_last_success_timestamp_seconds	Last successful restore drill
restic_restore_test_success	Whether the last restore drill passed

Alert response

ResticMetricsMissing

Prometheus can scrape node_exporter, but restic.prom is absent.

Check:

systemctl status restic-metrics.timer restic-metrics.service
ls -l /var/lib/node_exporter/textfile_collector/restic.prom
journalctl -u restic-metrics.service --since "2 hours ago"

Likely causes: backup-client did not converge, node_exporter textfile path is missing, or the metrics script exits before writing the file.

ResticMetricsStale

restic.prom exists but has not refreshed in over an hour.

Check:

systemctl list-timers 'restic-metrics*'
systemctl status restic-metrics.timer
journalctl -u restic-metrics.service --since "2 hours ago"
/usr/local/bin/restic-metrics

If a manual run fails, fix that error before restarting the timer.

ResticExpectedBackupMissing

A job is declared by a manifest, but no snapshot-count metric exists for that service.

Check:

systemctl status restic-backup-<service>.timer restic-backup-<service>.service
ls -la /var/backups/restic/<service>
RESTIC_REPOSITORY=/var/backups/restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Likely causes: repo initialization failed, the service path does not exist on that host, or restic-metrics cannot inspect the repo.

ResticBackupRunFailed

The last systemd run finished with a non-success result.

Check the unit and journal:

systemctl status restic-backup-<service>.service
journalctl -u restic-backup-<service>.service --since "24 hours ago"

For external jobs, also check the fetch hook under backups/external/. Failures can come from SSH, SOPS decrypt, Home Assistant token/API behavior, B2 copy, NAS copy, or restic prune.

ResticBackupTimerInactive

The service has an expected backup job, but the timer is not active.

Check:

systemctl list-timers 'restic-backup-*'
systemctl enable --now restic-backup-<service>.timer

If the timer file is missing, rerun Ansible for the backup-client role instead of creating an unmanaged timer.

ResticBackupStale

The latest local snapshot is older than 36 hours.

Check:

RESTIC_REPOSITORY=/var/backups/restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots
systemctl list-timers restic-backup-<service>.timer
journalctl -u restic-backup-<service>.service --since "48 hours ago"

If the last run succeeded but the snapshot age is still stale, verify the backup paths exist and restic is not backing up an empty or wrong path.

ResticBackupFailed

The local repo has zero snapshots. Treat this as critical for tier 1 and urgent for tier 2.

Check repo initialization and run one controlled backup after fixing path or credential issues:

RESTIC_REPOSITORY=/var/backups/restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots
systemctl start restic-backup-<service>.service
journalctl -u restic-backup-<service>.service -n 200 --no-pager

RestoreTestStale or RestoreTestFailed

The monthly restore drill has not succeeded recently, or the latest drill failed.

Check:

systemctl status restic-restore-test.timer restic-restore-test.service
journalctl -u restic-restore-test.service --since "45 days ago"
/opt/homelab/backups/restore-test.sh

If a restore failed for a specific service, continue with that service's section in restore procedures.

Copy targets

Tier 1 jobs normally copy to B2. Jobs with nas_copy: true also copy to Whrrr NFS cold storage. A failed copy should make the systemd backup unit non-success because copy operations run as ExecStartPost.

Check B2:

. /etc/restic/b2.env
RESTIC_REPOSITORY=b2:<bucket>:<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Check NAS:

mountpoint /mnt/infra-backups
RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Validation

After remediation:

1. Run /usr/local/bin/restic-metrics.
2. Confirm restic.prom mtime updates.
3. Confirm the Backup / Restic Triage dashboard shows the expected service.
4. Confirm Prometheus alert state resolves.
5. For real data-loss risk, run a scoped restore to /tmp/restore-<service> and follow restore procedures.