Skip to content

Check-in — 2026-07-31

TL;DR

The native OneUptime workflow failed concurrent correlation, but the replacement PostgreSQL-backed adapter is now live and has passed lifecycle, durability, reconciliation, notification, and restore gates. Production promotion is live: the routes admit allowlisted runner, node-availability, container-runtime, infra-capacity, and backup-freshness alerts while preserving direct Discord delivery. The failed workflow and duplicate incidents remain disabled, retained evidence.

Phase status

Area Status Evidence
OneUptime Live Pinned 11.7.3 stack remains healthy behind Authentik
Incident adapter Live Private API, worker, and dedicated PostgreSQL pass smoke
Native workflow Rejected Concurrent deliveries created duplicate incidents; workflow remains disabled
Adapter lifecycle Passed Concurrent firing, multi-fingerprint resolution, recurrence, replay, and OneUptime convergence passed
Native alert path Passed Prometheus firing and resolution advanced adapter webhook 9→11 and Discord 6→7
Recovery Passed Expired lease restart, reconciliation repair, and outage no-mutation gates passed
Data protection Passed Logical backup and isolated restore verified six correlations and seven outbox rows
Production promotion Live Real-shaped firing reached OneUptime and Discord through the promoted additive route
Node availability Passed Synthetic lifecycle and real sonarqube outage reached OneUptime while Discord remained active
Container and capacity Passed Both domains passed lifecycle, Alertmanager routing, Discord delivery, and dashboard provisioning
Backup freshness Passed Synthetic concurrency plus real warning, escalation, backup, and resolution reached one OneUptime incident

What shipped

  • Deployed the race-safe OneUptime incident adapter with transactional ingest, deterministic generations, a durable outbox, audited replay, and complete-set reconciliation.
  • Restricted connectivity to private Docker networks and least-privilege project credentials; no adapter service publishes a host or Traefik port.
  • Routed one fingerprint per Alertmanager webhook while preserving additive direct notification delivery.
  • Passed direct concurrency and recurrence acceptance after adopting and replaying the first failed OneUptime projection.
  • Passed a native Prometheus-to-Alertmanager lifecycle. Incident 1c9606bc-2d96-44a5-960d-6f4ad7c682a6 reached Identified and then Resolved; webhook counters advanced 9→11 and Discord advanced 6→7.
  • Proved reconciliation repair using two compatibility-test firings that had been missed by the webhook path; both later resolved after the absence grace.
  • Reclaimed an expired outbox lease after a worker restart and proved an Alertmanager ConnectError does not mutate incident state.
  • Created backup 20260731T221056Z and restored it in isolation with six correlations and seven outbox rows. The live queue ended with seven succeeded rows and no other states.
  • Promoted the merged route and accepted a real-shaped alert without a synthetic label. OneUptime incident 501d0337-1ea5-4031-b904-51c185c39bfb reached Resolved revision 2, both webhook and Discord notification counters advanced, and the queue ended with eight succeeded rows.
  • Corrected two deployment hazards exposed during promotion: stale adapter configuration selection during environment rendering and source-config mode 0640, which prevented Alertmanager's nobody user from reading its bind mount after recreation.
  • Deployed the first Stage 5 domain, NodeExporterUnavailable, with strict per-instance correlation and complete alert-schema labels.
  • Passed twenty-five concurrent node firings, an exact repeat, resolution, and recurrence. Generation one resolved at revision 27 as incident b47239fd-d092-4162-a91a-04fb955a72a0; generation two reached Identified as 85893b27-f8ff-43ce-a038-feeef4d6fc62.
  • Converged the real sonarqube outage as identified incident 3713a0ff-8384-4d12-9707-ace76975b4c4. Adapter webhook and Discord counters both advanced from zero to one, all failure counters remained zero, and all three node-domain projections reached succeeded.
  • Provisioned node and runner host availability in Alert Triage and verified the updated dashboard through Grafana's API.
  • Passed the container-runtime and infra-capacity lifecycle modes under acceptance key adapter-acceptance-1785550215, including concurrent deliveries, resolution, recurrence, and shared disk-threshold correlation.
  • Sent controlled ContainerRestarting and InfraServicesDiskPressure alerts through Alertmanager. Both created and resolved durable OneUptime incidents; both valid firings reached Discord with zero Discord failures.
  • Confirmed the capacity synthetic-identity guard rejects an unreserved test host. Its three expected webhook retries failed closed while the same firing still reached Discord, proving continue: true keeps direct delivery independent.
  • Verified Prometheus loaded the new rules and Grafana loaded the container and infra-services incident panels after restoring bind-mount readability.
  • Found that Komodo's later /opt/homelab checkout can undo Ansible's permission repair. Moved the durable a+rX repair to the repo on_pull hook so it runs after every checkout and before stack recreation.
  • Promoted ResticBackupStale and ResticBackupFailed with one canonical instance,backup_service fingerprint and a manifest-backed adapter allowlist. Prometheus loaded exactly one job="node" series for the live adapter repository.
  • Passed backup-freshness lifecycle acceptance under key adapter-acceptance-1785558865: twenty-five concurrent deliveries, escalation, independent threshold resolution, and recurrence all converged.
  • Observed the real empty adapter repository create OneUptime incident 059e9b62-6dd7-4436-a955-42fb5b46af0a at warning, then escalate the same generation to critical. Alertmanager inhibited only the matching stale warning while the critical alert remained active.
  • Ran the first controlled oneuptime-incident-adapter backup. Snapshot 5c2cfb58 preserved ten files and 143,908 bytes; refreshed metrics cleared both Prometheus and Alertmanager alerts, and the same incident resolved at revision 5 with a succeeded outbox projection.

Known gaps / drift

  • OneUptime 11.7.3's native workflow engine acknowledges the webhook before asynchronous execution and provides no unique correlation constraint or atomic find/create operation.
  • Concurrent runs 5dd191d8-31f4-4928-9b47-43e1fb451bd6 and 3c87ef89-fbe3-4905-9cbc-358c1345bf49 created incidents 0edc3107-ace1-44d1-8413-05d873d78dfd and 4703491b-0e6a-4373-807d-4f83c7d6d42c for the same correlation key.
  • The later multi-fingerprint, resolution, Alertmanager-route, and restart phases were intentionally not run through the rejected native workflow.
  • The native workflow and duplicate incidents must remain available as evidence; they are not production state and must not be deleted.
  • Production promotion covers four GitHub runner alert names, NodeExporterUnavailable, two container-runtime alerts, three infra-services capacity alerts, and two backup-freshness alerts. Remaining alert domains require their own schema, dashboard, and lifecycle acceptance.

What remains

Item Owner Priority
Migrate restore-test, patching, TLS, and platform-health domains incrementally Agent P2
Connect approval-gated runner diagnostics and recovery to incident timelines Agent P2
Finish lifecycle notifications, self-monitoring, and platform closeout Agent P2

Incident path

flowchart LR
  P[Prometheus] --> AM[Alertmanager]
  AM -->|"continue: true"| Direct[Discord]
  AM -->|"allowlisted domain fingerprints"| Adapter[Race-safe adapter]
  Adapter --> DB[(PostgreSQL + outbox)]
  DB --> Worker[Worker + reconciliation]
  Worker --> OneUptime[OneUptime incident API]
  AM -. "rejected" .-> Native[Disabled native workflow]