Backup alert triage

Use this runbook for restic backup and restore-test alerts. The first goal is to decide which layer is failing: metrics collection, systemd scheduling, fetch hooks, NAS restic snapshots, B2 copy targets, or restore validation.

Do not delete, move, or prune backup artifacts while triaging without explicit owner approval.

Entry points

Surface	Use
Backup / Restic Triage	Backup health overview and per-service drilldown
Alertmanager	Active backup alert groups and silences
Restore procedures	Per-service restore commands
DR from Zero	Full rebuild order

Scope

The backup-client role currently runs on Ansible docker_hosts except unprivileged Proxmox LXCs (proxmox_guests): today that means infra-services only. Phoenix and sonarqube stay in docker_hosts for other roles, but they cannot mount Whrrr /volume6/infra-backups from inside the CT, so in-guest restic is disabled until a host-side or idmap-safe path exists. Those guests rely on Proxmox vzdump / manual archive in the meantime.

ResticMetricsMissing therefore scopes to instance="infra-services" only (not phoenix/sonarqube). Backup manifests may still list phoenix and sonarqube for documentation; they are not applied by ansible-pull on those hosts.

Expected backup jobs from current manifests:

• Tier 1 on infra-services: external-authentik-db, external-saltbox-secrets, komodo, oneuptime, oneuptime-incident-adapter, paperless, and traefik.
• Tier 2 on infra-services: adguard, ara, external-haos-config, homepage, litellm, monitoring, paperless-stirling-bridge, and stirling-pdf. The Wazuh tier-2 job was removed with the stack.
• Tier 2 appliance-local jobs (manifest only until LXC NFS is fixed): phoenix/phoenix and sonarqube/sonarqube.

harbor-registry is documented in backups/external/harbor-registry.yaml, but it is not automatically discovered by backup-client today because it has no fetch_script and the host is not in docker_hosts.

The weekly restore-test timer runs only on infra-services. Each execution selects the next local tier-1 service manifest in sorted rotation: komodo, oneuptime, oneuptime-incident-adapter, paperless, or traefik. State lives in /var/lib/restic-restore-test/next-index and advances even when a drill fails so one bad repo cannot starve the others. With five candidates and a weekly cadence, each repo is exercised about every 35 days — matching the RestoreTestStale window. External tier-1 jobs and all tier-2 jobs remain outside this drill.

Metrics

The collector at /usr/local/bin/restic-metrics writes /var/lib/node_exporter/textfile_collector/restic.prom. node_exporter reports the textfile mtime with a file label that may be the full path, not just restic.prom.

Metric	Meaning
restic_backup_expected	Job expected from a backup manifest
restic_backup_copy_expected	Expected copy target, currently b2 or nas
restic_backup_last_success_timestamp_seconds	Last snapshot timestamp in the local repo
restic_backup_snapshot_count	Snapshot count in the local repo
restic_backup_last_size_bytes	Size of the latest local snapshot
restic_backup_last_run_result	Last systemd result label for the backup unit
restic_backup_last_run_exit_code	Last systemd ExecMainStatus
restic_backup_last_run_timestamp_seconds	Last backup unit completion time
restic_backup_last_run_duration_seconds	Last backup unit duration
restic_backup_timer_active	Whether the systemd timer is active
restic_backup_timer_next_run_timestamp_seconds	Next timer fire time
restic_restore_test_last_success_timestamp_seconds	Last successful restore drill
restic_restore_test_success	Whether the last restore drill passed

Alert response

ResticMetricsMissing

Prometheus can scrape node_exporter, but restic.prom is absent.

Check:

systemctl status restic-metrics.timer restic-metrics.service
ls -l /var/lib/node_exporter/textfile_collector/restic.prom
journalctl -u restic-metrics.service --since "2 hours ago"

If the file exists locally but the alert still fires, check Prometheus for the raw textfile label:

node_textfile_mtime_seconds{file=~"(.*/)?restic\\.prom"}

Likely causes: backup-client did not converge, node_exporter textfile path is missing, or the metrics script exits before writing the file.

ResticMetricsStale

restic.prom exists but has not refreshed in over an hour.

Check:

systemctl list-timers 'restic-metrics*'
systemctl status restic-metrics.timer
journalctl -u restic-metrics.service --since "2 hours ago"
/usr/local/bin/restic-metrics

If a manual run fails, fix that error before restarting the timer.

ResticExpectedBackupMissing

A job is declared by a manifest, but no snapshot-count metric exists for that service.

Check:

systemctl status restic-backup-<service>.timer restic-backup-<service>.service
ls -la /mnt/infra-backups/homelab-restic/<service>
RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Likely causes: repo initialization failed, the service path does not exist on that host, or restic-metrics cannot inspect the repo.

ResticBackupRunFailed

The last systemd run finished with a non-success result.

Check the unit and journal:

systemctl status restic-backup-<service>.service
journalctl -u restic-backup-<service>.service --since "24 hours ago"

For external jobs, also check the fetch hook under backups/external/. Failures can come from SSH, SOPS decrypt, Home Assistant token/API behavior, B2 copy, NFS mount issues, or restic prune.

Repository lock contention (exit 11)

repository is already locked / unable to create lock in backend does not by itself establish a stale lock. A live reader can hold a shared lock while forget --prune needs an exclusive lock. A successful main backup can therefore be followed by failed retention and a failed unit with ExecMainStatus=0. Post-steps run in order: a local retention failure prevents the later B2 copy.

The current template runs a best-effort primary unlock before the backup, not immediately before local retention, and another offsite unlock before B2 copy. Neither prevents a new reader from acquiring a lock later. Do not assume that another full backup will self-heal the failure.

1. Preserve the failing invocation's journal, snapshot IDs, lock ID, lock timestamp, host and PID. Identify whether backup, local retention, copy or offsite retention failed; main-process success alone is insufficient.
2. Check the lock-owning process on the recorded host. A later absent PID or empty lock list does not prove that the lock was stale at failure time. Metrics snapshots/stats readers also take shared locks; do not interrupt them merely to clear the failure.
3. Inspect primary and offsite snapshot metadata independently. For inspection-only snapshots / list locks, --no-lock --no-cache avoids adding a reader lock or local cache writes; it does not provide a transactionally stable view during concurrent maintenance. Never use it to bypass locking for writes or pruning. A copied snapshot's ID can differ: compare its originating ID and corroborating tree, timestamp, paths and host. Keep repository credentials on the host; never print environment files.
4. For transient contention, see the local-retention source fix below. Lock-acquisition retry preserves exclusive locking and failure on timeout; diagnose persistent contention separately.
5. If primary snapshots exist but required copies are missing, seek explicit owner approval for a scoped catch-up copy of those snapshots and verify destination metadata afterward. Do not rerun the whole backup, prune, force-unlock or reset the failed unit just to make its status green. These are separate reviewed actions.

Historical verification, 2026-10-27 11:29–11:30 UTC: OneUptime primary 4751b022 and adapter primary 5dd10199 existed, but their October 27 B2 copies were missing; latest B2 snapshots were October 26. Both jobs failed at local retention on fresh shared locks. Metrics overlap is plausible, not proven process attribution. No repository mutation or restore test occurred during that verification; see the dated backup evidence.

Current closure: explicitly owner-approved exact-snapshot copies recovered both in B2 as 9fb98052 and 28878252, respectively, with matching originating IDs and tree/time/paths/host. See the completed catch-up receipt. Only these two offsite gaps are closed; original failed-unit evidence remains untouched. This is not restore integrity, successful retention, scheduled-run acceptance or permission for further mutations.

Local-retention lock retry: source fix

PR #520 is merged as ea4acc8. The source fix updates only the first, local-retention ExecStartPost in the shared infra/ansible/roles/backup-client/templates/restic-backup.service.j2:

ExecStartPost=/usr/local/bin/restic forget --prune --retry-lock=5m {{ forget_args }}

Normal Ansible convergence renders this merged change for all managed backup jobs, not just OneUptime. Five minutes bounds lock-acquisition retry only, not total retention runtime. Exclusive locking, --prune, existing retention policies and the required-command failure/stop chain remain intact: failure to acquire the lock within that window still fails the unit and prevents later B2 copy. No --no-lock, new unlocks, error suppression or B2 flag changes are introduced.

Merged source is not live convergence or next scheduled-run acceptance. At the 11:55 preflight, the OneUptime and adapter live units still lacked --retry-lock=5m; normal convergence and next scheduled-run validation remain pending. The separate completed catch-up used copy-only flags, not a live unit alteration, and proves neither successful retention nor restore integrity. Its approval covered only those two snapshots; no unlock, prune, full backup, failed-unit reset, restart or Ansible apply occurred. Normal timers/convergence stay on, without forced apply; the edge guard, disabled deployment and #518 bridge maintenance hold are unchanged.

PostgresLogicalBackupRunFailed

This narrowly scoped page covers only restic-backup-paperless.service and restic-backup-litellm.service on infra-services, from job="node". A reported non-success systemd Result with metric value 1 must persist for five minutes. Missing/empty results, the collector's unknown fallback, and success do not match. The original service is preserved as backup_service before the alert sets service="backup". Its per-host/per-job run-failure correlation and inhibition key are separate from backup-freshness incidents.

ExecStartPre export failures can leave ExecMainStatus=0 and main-process completion timestamps zero or left over from an earlier run. Neither a current main timestamp nor a stale/empty repository is required: a fresh NAS snapshot must not hide a later B2 or cleanup failure. The existing collector actually uses ActiveEnterTimestamp / InactiveEnterTimestamp for its run timestamps, not ExecMainExitTimestamp; these are not reliable proof of this invocation's export. Allow for the independent 15-minute metrics timer plus the five-minute alert hold and normal Alertmanager grouping delay; this is not immediate failure detection.

The existing notify="page" route selects the Discord receiver. This does not add an adapter incident contract or broaden any ticket allowlist. The original ResticBackupRunFailed warning/ticket rule remains unchanged and unrouted; OPS-042 remains open. Source routing tests are not live delivery proof.

Preserve evidence and identify the failed stage

Use read-only inspection on the affected host; do not restart the backup simply to clear the alert. Preserve the original journal, invocation IDs, generation names and manifest/checksum evidence before seeking a reviewed recovery action. For example:

systemctl show restic-backup-paperless.service restic-backup-litellm.service \
  --no-pager --property=Id,Result,ExecMainStatus,ExecMainStartTimestamp,ExecMainExitTimestamp,InvocationID,ControlPID,ActiveState,SubState
journalctl -u restic-backup-paperless.service -u restic-backup-litellm.service \
  --since "24 hours ago" --no-pager -n 200
systemctl status restic-metrics.timer restic-metrics.service --no-pager
journalctl -u restic-metrics.service --since "2 hours ago" --no-pager -n 100
df -B1 /var/backups
df -i /var/backups

1. Export/pre-hook refusal: correlate the failing invocation with its journal. Check available bytes and inodes on the actual logical-generation filesystem, dump/catalog limits, generation count, and namespace validation. The fixed safety proposal below is a refusal policy, not permission to reclaim space.
2. NAS backup/retention or required B2 copy/retention failure: identify the exact failed step, mount/backend availability, credentials and repository-lock evidence. A successful NAS snapshot is not proof of B2 success or of a complete logical recovery set. Preserve local generations after downstream failure; repeated exports are not a repair for a failed copy.
3. Local generation cleanup refusal: verify ordering and invocation evidence against the effective unit, not just the checked-out template. A later ControlPID=0 is normal after completion; it cannot authorize a manual replay. Preserve incomplete cleanup markers and request review rather than resuming or forcing deletion. A content-change refusal is meaningful even if file size, inode and timestamps appear unchanged. Do not replace checksums/manifests to make the check pass. The content-stability repair in source uses original validated baselines and fresh reads; verify which revision is actually deployed. Repeated hashing can consume CPU/I/O: the cap-sized 14→7 prune projection is 3.5 GiB of Paperless dump reads or 24.5 GiB of LiteLLM dump reads, plus bounded metadata/header reads. These are logical bytes, not measured runtime; see the private operating contract linked from the repository README before changing timeouts or scheduling. No automatic retry or forced cleanup is authorized.
4. Collection ambiguity: the collector suppresses systemctl show errors and substitutes result="unknown"; that is missing evidence, not a confirmed failed backup (and is excluded from this page). It also substitutes [] when restic snapshots fails, potentially publishing zero snapshots and a zero last-success timestamp despite an existing backup. Diagnose collection, permissions, mounts and repository access before interpreting an empty-repo alert as data loss. Stale textfile data can keep an old result visible; check freshness and the journal. These collector limitations are not repaired here.

Fixed storage safety proposal for review

The bounded policy proposed for PR #474 is not evidence of deployed limits or owner acceptance:

• Keep the seven newest validated completed generations per service, by generation ordering, not seven days. The hard ceiling is 14 generation slots per service; at the ceiling new exports refuse without deleting data.
• Custom dumps are limited to 64 MiB for Paperless / 512 MiB for LiteLLM; producer catalog output is limited to 32 MiB. An oversize export fails and preserves bounded failed staging for review; these are not database-size claims.
• Before export and during streaming, filesystem available bytes must retain max(15% of total bytes, 8 GiB) + 16 GiB operational reserve, in addition to the prospective write/reservation. Available inodes must retain max(5% of total inodes, 100,000), plus prospective inode allocation. Unknown/invalid accounting fails closed. Shared workloads and restore staging still need a fresh capacity assessment; the reserve is not a quota or guarantee.
• Failed .staging-*, incomplete .deleting-*, foreign/unknown entries, malformed names, symlinks/hardlinks, corrupt manifests/checksums, or oversize existing generations/metadata block new exports. Preserve those residues and legacy outputs; seek review. Do not remove them, bypass limits, or use an emergency/forced prune to manufacture headroom.
• The post-backup pruner must run directly as the same systemd invocation's current ControlPID, in the named unit's activating/start-post phase, with the same INVOCATION_ID. Exactly one matching manifest generation must exist and be the newest. A newer, replaced or ambiguous generation causes refusal. It may run only after successful NAS backup/forget and all required B2 copy/forget steps (Paperless requires B2); a failed prior step blocks cleanup. Preserve at least the newest seven and the current-invocation generation. There is no automatic active-generation deletion, manual cleanup authorization, or permission to touch legacy dumps, PGDATA, or restic snapshots in this triage.

Normal reviewed apply/convergence, exact effective unit metadata and invocation ordering, real export/restore behavior, live failure-notification delivery and owner acceptance remain separate gates. Do not inject failures or send synthetic pages as part of this procedure, or treat a successful source test as acceptance.

ResticBackupTimerInactive

The service has an expected backup job, but the timer is not active.

Check:

systemctl list-timers 'restic-backup-*'
systemctl enable --now restic-backup-<service>.timer

If the timer file is missing, rerun Ansible for the backup-client role instead of creating an unmanaged timer.

ResticBackupStale

The latest local snapshot is older than 36 hours. This alert is scoped to jobs that emit restic_backup_expected on the same host; stray or retired repo directories can still appear in raw metrics but should not alert unless the job is expected there.

ResticBackupStale and ResticBackupFailed share one OneUptime incident per canonical instance,backup_service. The critical empty-repository threshold inhibits only the warning for that same job; unrelated backup incidents remain visible. The adapter accepts only host/job pairs declared by current backup manifests; synthetic acceptance uses the reserved adapter-acceptance-* job namespace.

Check:

RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots
systemctl list-timers restic-backup-<service>.timer
journalctl -u restic-backup-<service>.service --since "48 hours ago"

If the last run succeeded but the snapshot age is still stale, verify the backup paths exist and restic is not backing up an empty or wrong path.

ResticBackupFailed

The local repo has zero snapshots. Treat this as critical for tier 1 and urgent for tier 2. Like stale backups, this alert is scoped to expected jobs on that host.

Check repo initialization and run one controlled backup after fixing path or credential issues:

RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots
systemctl start restic-backup-<service>.service
journalctl -u restic-backup-<service>.service -n 200 --no-pager

Restore-test alerts

RestoreTestMetricsMissing means the canonical host has never emitted or has lost its restore-test metric. RestoreTestStale means the weekly drill has not succeeded recently, and RestoreTestFailed means the latest drill failed.

Both thresholds share one host-level OneUptime incident keyed by canonical instance. Restore drills rotate one tier-1 repository per run, so the last selected repository is diagnostic context (restore_service), not durable incident identity. The critical failure inhibits only the overdue warning on the same host.

Check:

systemctl status restic-restore-test.timer restic-restore-test.service
journalctl -u restic-restore-test.service --since "45 days ago"
/opt/homelab/backups/restore-test.sh

If a restore failed for a specific service, continue with that service's section in restore procedures.

Copy targets

Primary restic repos live on Whrrr NFS (/mnt/infra-backups/homelab-restic). Tier-1 (and explicit offsite: true) jobs also copy to B2. A failed B2 copy should make the systemd backup unit non-success because copy operations run as ExecStartPost.

Check B2:

. /etc/restic/b2.env
RESTIC_REPOSITORY=b2:<bucket>:<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Check NAS:

mountpoint /mnt/infra-backups
RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/<service> \
  RESTIC_PASSWORD_FILE=/etc/restic/password \
  restic snapshots

Validation

After remediation:

1. Run /usr/local/bin/restic-metrics.
2. Confirm restic.prom mtime updates.
3. Confirm the Backup / Restic Triage dashboard shows the expected service.
4. Confirm Prometheus alert state resolves.
5. For real data-loss risk, run a scoped restore to /tmp/restore-<service> and follow restore procedures.