Skip to content

ADR-003: OneUptime owns incident state

  • Status: Accepted
  • Date: 2026-07-29

Context

Prometheus, Alertmanager, Grafana, and Loki detect, route, and investigate operational problems successfully. The current workflow ends primarily in Discord or ntfy and does not retain durable acknowledgement, assignment, incident status, action history, or review evidence.

Alertmanager must remain the routing and silence authority. Prometheus and the Loki ruler must remain the alert-evaluation authorities. The added component must be self-hosted, open source, reproducible, and downstream of Alertmanager.

Keep Community Edition and OneUptime Community Edition were evaluated.

Decision

Deploy OneUptime Community Edition as the canonical incident-state system after capacity, backup, and correlation acceptance gates pass.

Use OneUptime for:

  • incident creation and deterministic correlation;
  • acknowledgement, assignment, mitigation, and resolution state;
  • incident and remediation timelines;
  • runbook, Grafana, Loki, Alertmanager, and GitHub Actions links;
  • incident lifecycle notification workflows;
  • post-incident evidence.

Do not use OneUptime to replace Prometheus metrics, Loki logs, Prometheus/Loki alert rules, Alertmanager routing and silences, Grafana dashboards, or the GitHub Actions/Ansible remediation control plane.

Rationale

OneUptime is actively maintained and its repository is Apache-2.0 licensed. Its self-hosted product includes incidents, workflows, runbooks, timelines, and API/webhook integration.

Keep has a strong alert-centric model, but its open-source edition is now community-maintained after the Elastic acquisition, the repository contains both MIT and commercial code, and released configuration-as-code support has gaps. Those risks are material for a new authoritative operational record.

OneUptime is heavier because PostgreSQL, Redis, and ClickHouse are required. This cost is accepted only with a measured resize of infra-services and resource limits that preserve the existing monitoring plane.

Consequences

Positive

  • Incidents have durable status, ownership, timelines, and review history.
  • Existing detection and notification continue during migration or OneUptime failure.
  • Correlation remains label- and fingerprint-based rather than opaque.
  • Remediation reuses guarded workflows instead of granting infrastructure credentials to the incident platform.

Negative

  • Community Edition OIDC is unavailable in OneUptime v11. Human access requires Authentik forward-auth followed by OneUptime local login.
  • The stack adds three stateful dependencies and requires additional compute, memory, disk, backup, and restore work.
  • A rollback after database migration requires data restoration, not just an image downgrade.
  • The supported deployment is not an incident-only product profile. Omitting the probe and AI agent requires explicit acceptance testing.

Guardrails

  • OneUptime is not the only critical notification path.
  • No production alerts are forwarded before an isolated restore succeeds.
  • No alert domain is promoted before repeat and resolved events update the same incident deterministically.
  • OneUptime receives no Docker socket, privileged mode, SSH key, SOPS key, or arbitrary command capability.
  • Disruptive actions require explicit human approval in GitHub protected environments.
  • Alertmanager silences remain separate from OneUptime acknowledgement.

Revisit conditions

Reconsider this decision if:

  • OneUptime Community Edition removes a core incident or workflow capability;
  • the incident-only deployment cannot stay within the approved host budget;
  • deterministic update and resolution cannot be made race-safe;
  • backups cannot restore incident and workflow history reliably;
  • a uniformly open-source, lighter platform demonstrably meets the same requirements with lower operational risk.