Skip to content

Check-in — 2026-08-28 OneUptime alert recovery

TL;DR

The active OneUptime queue is back to normal after eight transient HTTP 502 dead letters were audited and replayed. Central bulk-media capacity is now dashboard-only because expansion and deletion are not viable remediations; the separate Whrrr DSM push requires one owner task to disable. Graylog convergence recovered from a corrupt Git checkout, and the same failure will self-heal after this change reaches hosts.

Phase status

Item Status
OneUptime adapter dead letters Eight replayed; all succeeded
Transient downstream retry policy Bounded-delay retries no longer exhaust
Bulk media capacity policy Prometheus dashboard-only; five OneUptime incidents resolved
Whrrr standalone capacity push Repository default disabled; live DSM task awaits owner disable
Graylog ansible-pull Corrupt clone quarantined; fresh apply succeeded
Coordinated patching All 12 targets succeeded; Phoenix and SonarQube incidents resolved
Proxmox NVMe latency SMART passed; low live latency; false counter spike now needs busy-time corroboration

What shipped

  • Audited replay recovered all eight stranded projections without deleting adapter or OneUptime history.
  • The adapter distinguishes known transient OneUptime failures from permanent failures and unexpected exceptions. Transient HTTP and transport errors keep retrying while queue-age metrics expose an ongoing outage.
  • The static NodeFilesystemBulkPressure rule and its OneUptime route were removed. Alert triage records the owner policy and keeps raw capacity visible in Grafana.
  • The independent Whrrr script now defaults to disabled, and the Synology capacity runbook records the non-destructive DSM task-disable procedure.
  • Graylog's corrupt /var/lib/ansible-pull/homelab clone was moved to a timestamped quarantine and replaced successfully. Pull units now run a Git integrity preflight that performs that recovery automatically.
  • The coordinated patch run exposed and recovered an apt/ansible-pull deadlock on recordurbate. Future runs pause pull timers around package work and restore them even when patching fails.
  • Patch success time is generated inside the target-side atomic metric write; the corrected Phoenix and SonarQube metrics closed both stale incidents.
  • Proxmox nvme0n1 passed SMART with no media or integrity errors, while live iostat showed about 20% utilization and millisecond latency. The alert now requires over 80% device busy time so corrupted latency counters do not fire without real saturation.
  • Discord webhook post tasks now suppress module arguments from logs.

Known gaps / drift

  • The live monitoring and adapter files were deployed ahead of merge to stop noise and prevent another transient dead-letter cycle. Merge is required so GitOps no longer sees those files as drift.
  • Graylog's quarantined checkout remains retained for evidence until the owner chooses to remove it; it is not application or backup data.
  • Bulk volume headroom remains low by design. Grafana remains the visibility surface, but no alert claims that buying disks or deleting retained data is an available remediation.
  • Whrrr's root-owned DSM scheduled task is outside Alertmanager and remains live until the owner disables it in Task Scheduler. The agent's dedicated key is not authorized on DSM, so this could not be changed over SSH.
  • The current Discord patch webhook URL appeared in historical orchestrator journal output before the logging guard was added. Treat it as exposed and rotate it through the existing SOPS notification secret.

What remains

Who Work Priority
Owner Review and merge the alert-reliability PR P0
Owner Disable Whrrr's 30-minute hotfix-syno-capacity.sh DSM task P0
Owner Rotate the patch Discord webhook and redeploy SOPS notification secrets P0
Agent Confirm host convergence installs the ansible-pull preflight after merge P1 
flowchart LR
  prom[Prometheus] --> am[Alertmanager]
  am --> adapter[OneUptime adapter]
  adapter --> outbox[(Durable outbox)]
  outbox --> retry[Bounded transient retry]
  retry --> oneuptime[OneUptime]
  bulk[Bulk capacity metrics] --> grafana[Grafana only]
  corrupt[Corrupt pull clone] --> quarantine[Timestamped quarantine]
  quarantine --> reclone[Clean ansible-pull clone]
  patch[Coordinated patch] --> pause[Pause pull timers]
  pause --> apt[Atomic package + metric work]
  apt --> restore[Restore pull timers]