Operations and incident platform

Status: target architecture; Stages 0–5 implemented per operations platform migration. Stages 6–8 remain.

Executive assessment

The repository already has a useful observability plane on infra-services:

• Prometheus evaluates infrastructure and application alerts.
• Alertmanager groups and routes notify=page alerts to Discord, allowlisted service/severity/alert-name combinations to the incident adapter, and critical patching alerts to ntfy. Unmatched alerts reach blackhole.
• Grafana provisions dashboards from Git and uses Prometheus and Loki.
• Loki and Alloy retain 30 days of logs from infra-services and runner hosts.
• Inventory generates node-exporter targets.
• GitHub Actions and Ansible provide guarded runner maintenance and diagnostics.
• Traefik and Authentik protect human-facing monitoring interfaces.
• SOPS-backed renderers keep runtime secrets out of Git.

The principal gap is operational state. Alertmanager tracks delivery, grouping, silences, and currently active alerts, but it is not an incident record system. Discord and ntfy messages do not provide durable acknowledgement, assignment, timelines, or review data.

The existing components remain authoritative in their current domains. OneUptime Community Edition adds incident state and workflow history; it does not replace Prometheus, Alertmanager, Grafana, Loki, or the remediation control plane.

Current gaps

Stages 0–5 of the operations platform migration are accepted (adapter live; domain rollout through monitoring-platform as of 2026-08-01). Remaining program gaps:

• Approval-gated runner remediation is not yet connected to OneUptime timelines (Stage 6).
• OneUptime lifecycle notifications have not cut over from dual-write/pilot to production destinations (Stage 7).
• The independent host heartbeat uses Healthchecks.io + ntfy outside Prom/AM (monitoring dead-man). It measures host/timer/outbound-network liveness, not monitoring-plane health; a healthy host can keep pinging with both monitoring processes stopped. Prometheus self-health rules require a working evaluation and delivery path to notify.
• continue: true evaluates later sibling routes, not universal failover. Discord requires notify=page; ntfy requires critical service=os-patching. Allowlisted runner notify=ticket alerts depend on the adapter without a direct Discord fallback. OneUptime is not yet the sole lifecycle notifier.
• Alert noise, frequency, duration, and flapping summaries still need a decision-oriented closeout dashboard pass.

Decision

Use OneUptime Community Edition as the incident and response layer.

OneUptime was selected over Keep Community Edition because OneUptime is actively maintained and uniformly Apache-2.0 licensed. Keep remains functionally capable, but its open-source edition is community-maintained after the Elastic acquisition, the repository mixes MIT and commercial code, and released configuration-as-code support has material gaps.

OneUptime is deployed only after capacity and restore gates pass. Version 11.7.3 is the researched baseline; deployment pins the exact application and dependency versions or digests.

Responsibilities and state ownership

Concern	Owner	Durable state
Metrics and alert series	Prometheus	Prometheus TSDB, 30-day baseline
Logs and log alert source data	Loki	Loki TSDB, 30-day baseline
Alert evaluation	Prometheus and Loki ruler	Version-controlled rules
Grouping, inhibition, routing, silences	Alertmanager	Alertmanager state volume
Incident status, acknowledgement, assignment	OneUptime	PostgreSQL
Incident and remediation timeline	OneUptime	PostgreSQL
Dashboards and investigation	Grafana	Provisioned JSON plus Grafana DB
Direct emergency notification	Alertmanager	Receiver configuration in Git
Incident lifecycle notification	OneUptime	Workflows plus provider secrets
Remediation execution	GitHub Actions and Ansible	GitHub run logs and artifacts
Infrastructure inventory	Inventory YAML	Git

Prometheus remains the sole source of truth for whether a metric alert is firing. A OneUptime acknowledgement does not silence or resolve an Alertmanager alert. Silence creation is a separate, explicitly permissioned action.

Data flow

flowchart LR
  Sources[Exporters, applications, checks] --> Prometheus
  Prometheus --> Alertmanager
  LokiRuler[Loki ruler] --> Alertmanager
  Alertmanager -->|matching page or critical patching route| DirectNotify[Discord and ntfy]
  Alertmanager -->|allowlisted adapter routes| OneUptime[OneUptime incidents]
  Prometheus --> Grafana
  Loki --> Grafana
  OneUptime --> Investigation[Grafana, Loki, and runbooks]
  OneUptime --> LifecycleNotify[Lifecycle notifications]
  OneUptime --> GitHubActions[Approved GitHub Actions]
  GitHubActions --> Ansible[Bounded Ansible actions]
  GitHubActions --> OneUptime

Alertmanager forwards only explicitly allowlisted service/severity/alert-name combinations to the adapter. send_resolved: true requests recovery delivery. Direct Discord and ntfy receivers do not depend on OneUptime for already-matching alerts, but they still depend on Alertmanager and their providers. Receiver selection is not proof of delivery. See the current receiver matrix and rollback boundary: removing an adapter route leaves incident-only tickets without notifications unless a fallback is separately reviewed or the owner accepts a bounded loss window.

Incident identity and lifecycle

Each Alertmanager alert keeps its native fingerprint as the alert identity. Repeated firing events update the same alert record.

Incidents use a versioned deterministic correlation key derived from the alert contract. The default dimensions are:

v1/environment/service/component/category/<domain-dimension>

The domain dimension is explicit, for example:

• runner_group for runner-capacity incidents;
• storage_pool for storage incidents;
• backup_target for backup incidents;
• instance or host for host-scoped availability incidents;
• repository and workflow for repository automation incidents.

Alertmanager groupKey is retained for traceability but is not the only long-term incident identifier because routing configuration can change it.

An incident closes only after every attached alert fingerprint is resolved. Partial group recovery updates the timeline and leaves the incident open.

Authentication boundaries

Human access

oneuptime.infra.realemail.app is LAN/VPN only. Browser pages pass through the local Traefik instance and authentik@file. /api and /identity use a higher-priority router with no Authentik middleware; OneUptime's own credentials authorize those calls, which is what the On-Call app requires. Community Edition on this stack does not use OIDC, so browser users still complete Authentik and then OneUptime login. A direct backend-network request must not bypass OneUptime authorization.

Machine access

Alertmanager and GitHub Actions use separate, least-privilege application tokens. Machine routes are exact paths with rate limits and source restrictions; they are not covered by a broad Authentik bypass.

OneUptime receives none of the following:

• Docker socket access;
• privileged mode;
• host filesystem mounts;
• the SOPS age key;
• patch-controller or operator SSH keys;
• runner registration or operator GitHub App credentials;
• arbitrary command execution.

Network boundaries

The OneUptime stack has a private application network for the application, PostgreSQL, Redis, ClickHouse, and its ingress proxy. Only the ingress proxy joins the external traefik network.

Alertmanager delivery prefers a private, explicitly shared ingestion network. If a machine route is required through Traefik, it is limited to the exact webhook path and protected by both a OneUptime token and edge controls.

No OneUptime container publishes a host port. Adding OneUptime does not add a WAN port forward or Cloudflare Tunnel.

Storage and capacity

OneUptime's supported Compose deployment requires PostgreSQL, Redis, and ClickHouse even when duplicate telemetry ingestion is disabled. The current infra-services allocation of 4 vCPU, 16 GiB RAM, and 100 GiB disk is therefore not sufficient.

Before deployment:

1. Measure live VM, hypervisor RAM, and thin-pool headroom.
2. Agree the final allocation with the owner.
3. Resize VM 123 and its guest filesystem in a maintenance window.
4. Update inventory only after reality changes.
5. Verify at least 20 percent disk free and sufficient idle memory.

The planning target is approximately 12 vCPU, 32 GiB RAM, and 200–400 GiB disk. The final disk allocation must account for current thin-pool headroom and OneUptime's actual incident-only growth.

Backup requirements

OneUptime is tier 1 once it becomes the incident record:

• PostgreSQL logical backup for incidents, users, workflows, and timelines;
• ClickHouse-consistent backup even while telemetry ingestion is disabled;
• SOPS-protected preservation of ENCRYPTION_SECRET and service credentials;
• exact Compose configuration and image digests in Git;
• off-host restic copy and an isolated restore drill.

Redis is cache and queue state and is restored empty.

Prometheus and Loki remain retention-bounded tier-2 telemetry unless a later owner decision promotes their history. Alertmanager state is included in monitoring recovery so active silences survive a host loss.

Failure modes

Failure	Detection	Degraded behavior
OneUptime unavailable	Readiness probe and container alerts, where configured	Only matching page/critical-patching routes retain direct destinations; incident-only tickets have no direct fallback
PostgreSQL unavailable	Readiness and database health checks	No incident mutations; only matching direct routes remain independent
Redis or workers unavailable	Queue and workflow health checks	Incidents may persist but workflows pause
ClickHouse unavailable	Analytics database health check	OneUptime readiness may fail; telemetry remains in Prometheus/Loki
Alertmanager unavailable	Prometheus may observe scrape failure, but notification through the failed Alertmanager can be lost	Host heartbeat can remain healthy; OneUptime cannot compensate for lost routing
Prometheus unavailable	Independent monitoring-chain watchdog acceptance remains pending; host heartbeat is insufficient	Metric alerts cannot evaluate; host heartbeat can keep pinging
Authentik unavailable	External SSO probe	Human UI is unavailable; machine ingress remains independent
GitHub Actions unavailable	Workflow dispatch and queue alerts	Investigation continues; remediation is manual
Notification provider failure	Delivery canary and workflow status	Incident console remains authoritative

An end-to-end monitoring watchdog must terminate outside the Prometheus/Alertmanager failure domain and separately prove evaluation and delivery loss detection. The existing independent host heartbeat is not that watchdog. Critical severity alone does not imply an independent delivery path: some critical alerts intentionally remain tickets. Changing that policy requires owner review.

Upgrade and rollback strategy

OneUptime releases frequently and applies database migrations at startup.

Upgrade:

1. Review release and security notes.
2. Test the exact target version against a restored dataset.
3. Take and verify PostgreSQL and ClickHouse backups.
4. Pull pinned images and record digests.
5. Stop ingress and application workers.
6. Start the new application and allow migrations to finish.
7. Test readiness, login, incident history, firing/resolved correlation, links, and notification/remediation workflows.

Rollback restores the previous PostgreSQL and ClickHouse backups and the prior exact images together. Reverting an image tag without restoring migrated data is not a rollback.

Implementation sequence

1. Land this architecture and the alert contract.
2. Add semantic CI and correct existing rule defects.
3. Validate and resize infra-services.
4. Deploy OneUptime as an authenticated shadow stack.
5. Prove backup and restore before alert ingestion.
6. Implement the GitHub runner correlation adapter after the native workflow concurrency gate failed.
7. Roll out alert domains and operational dashboards incrementally.
8. Connect approval-gated runner remediation.
9. Migrate lifecycle notifications and add platform self-monitoring.

See operations platform migration for gates and rollback points.

Known risks and deferred work

• Community Edition requires a second local login behind Authentik because OIDC is an enterprise feature.
• The five-service incident-only deployment omits OneUptime's probe and AI containers and must be retested on every upgrade.
• Native workflow find/create behavior failed duplicate safety on 2026-07-31: two concurrent deliveries created two incidents for one correlation key. The workflow remains disabled. The race-safe incident adapter replaced it and now accepts allowlisted production alerts.
• The monitoring host runs several stateful services. Capacity is a hard gate, not a documentation exercise.
• Synology, UPS, and external probes need additional exporters before their domains can be represented completely.