Skip to content

Check-in — 2026-08-01

TL;DR

Stage 5 alert-domain rollout is complete: restore, Komodo, patching, ansible-pull, TLS, and monitoring-platform (Alloy/Loki) all have live OneUptime lifecycle acceptance. Remaining work is Stage 6+ (runner remediation, dead-man coverage, platform closeout) and optional deeper restore coverage.

Phase status

Area Status Evidence
Restore timer ownership Passed Enabled and active only on infra-services; units absent on phoenix and sonarqube
Restore rule identity Passed One canonical job="node" pending/firing series; missing and failed signals clear
Synthetic lifecycle Passed Twenty-five concurrent deliveries, threshold correlation, resolution, and recurrence converged
Real lifecycle Passed Overdue incident identified, controlled Komodo restore passed, and the same incident resolved
Direct delivery Preserved Adapter route remains additive with continue: true
Komodo configuration Passed Drift and detector-health fingerprints correlated for the canonical infra-services deployment
Patch host health Passed Stale patch and reboot-pending fingerprints correlated per inventory-backed host
Patch orchestrator Passed Independent infra-services lifecycle converged through recurrence
Ansible pull Passed Independent per-host lifecycle converged through recurrence
Critical route delivery Passed Discord increased by one; adapter and ntfy webhooks increased by two with zero failures
TLS certificates Passed TLSCertExpiringSoon correlated per Traefik cn; live gate after stateChangedAt clamp
Monitoring platform Passed Alloy unavailable lifecycle accepted; Loki discards schema-complete with independent key

What shipped

  • Converged backup-client restore ownership across all three docker_hosts. restic-restore-test.timer remains enabled and active on infra-services; phoenix and sonarqube have no restore-test unit files.
  • Loaded RestoreTestMetricsMissing, RestoreTestStale, and RestoreTestFailed with complete alert-schema labels and one host-scoped correlation key.
  • Passed restore-health lifecycle acceptance under key adapter-acceptance-1785566354, including twenty-five concurrent deliveries, partial threshold resolution, full resolution, and recurrence.
  • Observed the real overdue drill create warning incident c126c8bf-7604-4fbe-b8f3-a9434c66c588 as generation 1.
  • Ran the controlled production restore drill against Komodo snapshot 4ffc085a. Restic restored 309 files, checked all 11 repository snapshots, and reported no errors.
  • Verified the refreshed canonical metric reported restic_restore_test_success{instance="infra-services",service="komodo"} 1. Prometheus and Alertmanager cleared the restore alert, and the same OneUptime incident resolved at revision 2 with a succeeded outbox projection.
  • Loaded the six migrated patching, ansible-pull, and Komodo rules from merged commit 0af18a3, with canonical service, component, category, and correlation labels visible through the live Prometheus rules API.
  • Passed focused lifecycle gates for komodo-config, patch-host-health, patch-orchestrator, and ansible-pull. Each gate proved 25 concurrent deliveries, resolution, successful outbox projection, and recurrence.
  • Sent controlled critical route alert adapter-acceptance-route-1785574986 through Alertmanager. Discord delivery increased from 5 to 6, webhook delivery increased from 0 to 2 with no failures, and OneUptime incident b1eaa1d2-c5d4-4de8-a6de-81a74f770742 resolved at revision 2.
  • Ran approved coordinated OS patch cycle via homelab-patch-orchestrate.service (exit 0). Cleared host-level OSPatchStale after fixing phoenix textfile success epoch.
  • Migrated TLSCertExpiringSoon into OneUptime (tls-certificate domain). Live acceptance passed after clamping adapter stateChangedAt when synthetic resolve timestamps precede generation create time (#212, #213).
  • Migrated CentralAlloyUnavailable and LokiIngestionErrors into monitoring-platform (#215). Retired duplicate PrometheusStoragePressure. Live Alloy acceptance passed as adapter-acceptance-1785646067.

Known gaps / drift

  • The drill rotates five local tier-1 service manifests weekly: komodo, oneuptime, oneuptime-incident-adapter, paperless, and traefik.
  • External tier-1 jobs and every tier-2 repository remain outside restore-test coverage.
  • Coordinated OS patch cycle ran evening PDT (homelab-patch-orchestrate exit 0; Discord summary 8 patched / 3 rebooted). Prometheus now shows fresh homelab_patch_last_success_unixtime on all scraped hosts and zero OSPatchStale / PatchRebootPending / AnsiblePullStale.
  • phoenix patch.prom was rewritten at success time with a stale ansible_date_time.epoch (~27 days old) despite a correct host clock and file mtime — manually corrected live; role fix uses date +%s on the target. phoenix boot remains Aug 1 18:52 (orchestrator labeled patched_rebooted but guest did not reboot in the wave window).
  • Unprivileged LXC NFS restic on Prox guests remains skipped (#210); rely on vzdump / host-side backups until a supported path exists.
  • Alertmanager/Grafana/Prometheus self-health and external dead-man coverage remain incomplete (must terminate outside the Prom/AM failure domain).
  • Dedicated service-health rules beyond container-runtime are optional.

What remains

Item Owner Priority
External dead-man / monitoring-plane self-health outside Prom/AM Agent P2
Design dedicated service-health rules if needed beyond container-runtime Agent P3
Connect approval-gated runner diagnostics and recovery to timelines Agent P2
Finish lifecycle notifications, self-monitoring, and platform closeout Agent P2

Accepted incident paths

flowchart LR
  Timer[infra-services restore timer] --> Drill[Rotated tier-1 restore drill]
  Drill --> Metrics[Node exporter textfile metrics]
  Metrics --> Prometheus
  Prometheus --> Alertmanager
  Alertmanager -->|"continue: true"| Direct[Discord]
  Alertmanager --> Adapter[Race-safe adapter]
  Adapter --> OneUptime[One host-scoped incident]
  Drill -->|"successful Komodo restore"| Resolved[Incident resolved]
  Patch[Patch and ansible metrics] --> Prometheus
  Komodo[Komodo secret propagation metrics] --> Prometheus
  TLS[Traefik cert metrics] --> Prometheus
  Mon[Alloy and Loki scrapes] --> Prometheus
  OneUptime --> History[Retained lifecycle history]