Skip to content

Check-in — 2026-07-31

TL;DR

OneUptime remains healthy, backed up, and isolated from production alert traffic. The synthetic native-workflow pilot proved sequential create and repeat behavior but failed deterministic correlation under concurrent delivery. The workflow is disabled, evidence is retained, and a race-safe adapter is now required before runner alerts can enter the incident platform.

Phase status

Area Status Evidence
OneUptime shadow stack Live Pinned 11.7.3 application, ingress, PostgreSQL, Redis, and ClickHouse remain healthy
Data protection Verified Isolated PostgreSQL and ClickHouse restore and tier-1 restic copies passed
Alertmanager route Shadow only Route requires synthetic="true" and keeps continue: true
Native sequential lifecycle Passed One firing created one incident; repeated firing updated the same incident
Native concurrent correlation Failed Two simultaneous deliveries created two incidents for one correlation key
Production runner ingestion Blocked Workflow disabled; real alerts remain on direct Discord and ntfy paths
Race-safe adapter Required Must pass durable enqueue, uniqueness, replay, reconciliation, and concurrency gates

What shipped

  • Deployed and verified the five-service OneUptime shadow stack behind Traefik and Authentik.
  • Proved tier-1 backup, isolated restore, and post-restore startup before alert ingestion.
  • Standardized runner labels, annotations, correlation keys, Prometheus tests, and the additive synthetic-only Alertmanager route.
  • Built the disabled native workflow and an executable synthetic lifecycle helper in the runner pilot runbook.
  • Corrected OneUptime's unsafe whole-object interpolation boundaries after two retained pre-gate workflow failures.
  • Passed sequential incident creation and repeat idempotency:
  • workflow run 85f1e1e5-bee9-4bf5-a167-f8d34932a64f created incident 3403ae6e-23d7-4929-a4e0-b026975a9c63;
  • workflow run d1336e2c-5df0-4a8c-b291-c25db7e87b51 updated the same incident and advanced Last Seen At.
  • Exercised concurrent creation and retained both successful workflow runs and duplicate incidents as failure evidence.

Known gaps / drift

  • OneUptime 11.7.3's native workflow engine acknowledges the webhook before asynchronous execution and provides no unique correlation constraint or atomic find/create operation.
  • Concurrent runs 5dd191d8-31f4-4928-9b47-43e1fb451bd6 and 3c87ef89-fbe3-4905-9cbc-358c1345bf49 created incidents 0edc3107-ace1-44d1-8413-05d873d78dfd and 4703491b-0e6a-4373-807d-4f83c7d6d42c for the same correlation key.
  • The later multi-fingerprint, resolution, Alertmanager-route, and restart phases were intentionally not run after the concurrency stop condition.
  • The native workflow and duplicate incidents must remain available as evidence; they are not production state and must not be deleted.

What remains

Item Owner Priority
Implement the narrow race-safe incident adapter Agent P1
Repeat create, update, concurrent create/resolve, replay, and reconciliation acceptance through the adapter Agent P1
Review and merge the adapter milestone PR Owner P1 after implementation
Remove synthetic="true" only after adapter acceptance passes Agent P1 after acceptance
Continue domain dashboards, remediation, notifications, and self-monitoring Agent P2 after runner promotion

Incident path

flowchart LR
  AM[Alertmanager] -->|"synthetic only"| Native[OneUptime native workflow]
  Native -->|"sequential"| Incident[One incident]
  Native -->|"concurrent"| Duplicate[Duplicate incidents]
  Duplicate --> Stop[Workflow disabled]
  AM -. "production blocked" .-> Adapter[Race-safe adapter]
  Adapter -->|"durable + unique"| OneUptime[OneUptime incident API]