Skip to content

Operations and incident platform

Status: target architecture; implementation is staged behind the gates in operations platform migration.

Executive assessment

The repository already has a useful observability plane on infra-services:

  • Prometheus evaluates infrastructure and application alerts.
  • Alertmanager groups and routes warning and critical alerts to Discord, with a separate ntfy route for critical patching alerts.
  • Grafana provisions dashboards from Git and uses Prometheus and Loki.
  • Loki and Alloy retain 30 days of logs from infra-services and runner hosts.
  • Inventory generates node-exporter targets.
  • GitHub Actions and Ansible provide guarded runner maintenance and diagnostics.
  • Traefik and Authentik protect human-facing monitoring interfaces.
  • SOPS-backed renderers keep runtime secrets out of Git.

The principal gap is operational state. Alertmanager tracks delivery, grouping, silences, and currently active alerts, but it is not an incident record system. Discord and ntfy messages do not provide durable acknowledgement, assignment, timelines, or review data.

The existing components remain authoritative in their current domains. OneUptime Community Edition adds incident state and workflow history; it does not replace Prometheus, Alertmanager, Grafana, Loki, or the remediation control plane.

Current gaps

  • Alert rules do not yet share a complete service, owner, environment, and category schema.
  • Alertmanager inhibition compares alertname, so warning and critical rules with different names are not inhibited as intended.
  • Prometheus and Alertmanager semantic checks are optional locally and absent from CI.
  • A TLS rule refers to a metric name that Traefik does not expose.
  • Resolved alert history is bounded by Prometheus retention and is not an incident timeline.
  • Monitoring-plane health and external dead-man coverage are incomplete.
  • Discord and ntfy are acting as the operator's primary queue.
  • Alert noise, frequency, duration, and flapping are not summarized in a decision-oriented dashboard.

Decision

Use OneUptime Community Edition as the incident and response layer.

OneUptime was selected over Keep Community Edition because OneUptime is actively maintained and uniformly Apache-2.0 licensed. Keep remains functionally capable, but its open-source edition is community-maintained after the Elastic acquisition, the repository mixes MIT and commercial code, and released configuration-as-code support has material gaps.

OneUptime is deployed only after capacity and restore gates pass. Version 11.7.3 is the researched baseline; deployment pins the exact application and dependency versions or digests.

Responsibilities and state ownership

Concern Owner Durable state
Metrics and alert series Prometheus Prometheus TSDB, 30-day baseline
Logs and log alert source data Loki Loki TSDB, 30-day baseline
Alert evaluation Prometheus and Loki ruler Version-controlled rules
Grouping, inhibition, routing, silences Alertmanager Alertmanager state volume
Incident status, acknowledgement, assignment OneUptime PostgreSQL
Incident and remediation timeline OneUptime PostgreSQL
Dashboards and investigation Grafana Provisioned JSON plus Grafana DB
Direct emergency notification Alertmanager Receiver configuration in Git
Incident lifecycle notification OneUptime Workflows plus provider secrets
Remediation execution GitHub Actions and Ansible GitHub run logs and artifacts
Infrastructure inventory Inventory YAML Git

Prometheus remains the sole source of truth for whether a metric alert is firing. A OneUptime acknowledgement does not silence or resolve an Alertmanager alert. Silence creation is a separate, explicitly permissioned action.

Data flow

flowchart LR
  Sources[Exporters, applications, checks] --> Prometheus
  Prometheus --> Alertmanager
  LokiRuler[Loki ruler] --> Alertmanager
  Alertmanager --> DirectNotify[Discord and ntfy fail-safe]
  Alertmanager --> OneUptime[OneUptime incidents]
  Prometheus --> Grafana
  Loki --> Grafana
  OneUptime --> Investigation[Grafana, Loki, and runbooks]
  OneUptime --> LifecycleNotify[Lifecycle notifications]
  OneUptime --> GitHubActions[Approved GitHub Actions]
  GitHubActions --> Ansible[Bounded Ansible actions]
  GitHubActions --> OneUptime

Alertmanager forwards only actionable warning and critical alerts during the initial rollout. send_resolved: true preserves the lifecycle. Existing Discord and ntfy receivers continue independently while OneUptime is piloted.

Incident identity and lifecycle

Each Alertmanager alert keeps its native fingerprint as the alert identity. Repeated firing events update the same alert record.

Incidents use a versioned deterministic correlation key derived from the alert contract. The default dimensions are:

v1/environment/service/component/category/<domain-dimension>

The domain dimension is explicit, for example:

  • runner_group for runner-capacity incidents;
  • storage_pool for storage incidents;
  • backup_target for backup incidents;
  • instance or host for host-scoped availability incidents;
  • repository and workflow for repository automation incidents.

Alertmanager groupKey is retained for traceability but is not the only long-term incident identifier because routing configuration can change it.

An incident closes only after every attached alert fingerprint is resolved. Partial group recovery updates the timeline and leaves the incident open.

Authentication boundaries

Human access

oneuptime.infra.realemail.app is LAN/VPN only and passes through the local Traefik instance and authentik@file. OneUptime v11 Community Edition does not include OIDC, so OneUptime local authentication remains enabled behind Authentik. A direct backend-network request must not bypass OneUptime authorization.

Machine access

Alertmanager and GitHub Actions use separate, least-privilege application tokens. Machine routes are exact paths with rate limits and source restrictions; they are not covered by a broad Authentik bypass.

OneUptime receives none of the following:

  • Docker socket access;
  • privileged mode;
  • host filesystem mounts;
  • the SOPS age key;
  • patch-controller or operator SSH keys;
  • runner registration or operator GitHub App credentials;
  • arbitrary command execution.

Network boundaries

The OneUptime stack has a private application network for the application, PostgreSQL, Redis, ClickHouse, and its ingress proxy. Only the ingress proxy joins the external traefik network.

Alertmanager delivery prefers a private, explicitly shared ingestion network. If a machine route is required through Traefik, it is limited to the exact webhook path and protected by both a OneUptime token and edge controls.

No OneUptime container publishes a host port. Adding OneUptime does not add a WAN port forward or Cloudflare Tunnel.

Storage and capacity

OneUptime's supported Compose deployment requires PostgreSQL, Redis, and ClickHouse even when duplicate telemetry ingestion is disabled. The current infra-services allocation of 4 vCPU, 16 GiB RAM, and 100 GiB disk is therefore not sufficient.

Before deployment:

  1. Measure live VM, hypervisor RAM, and thin-pool headroom.
  2. Agree the final allocation with the owner.
  3. Resize VM 123 and its guest filesystem in a maintenance window.
  4. Update inventory only after reality changes.
  5. Verify at least 20 percent disk free and sufficient idle memory.

The planning target is approximately 12 vCPU, 32 GiB RAM, and 200–400 GiB disk. The final disk allocation must account for current thin-pool headroom and OneUptime's actual incident-only growth.

Backup requirements

OneUptime is tier 1 once it becomes the incident record:

  • PostgreSQL logical backup for incidents, users, workflows, and timelines;
  • ClickHouse-consistent backup even while telemetry ingestion is disabled;
  • SOPS-protected preservation of ENCRYPTION_SECRET and service credentials;
  • exact Compose configuration and image digests in Git;
  • off-host restic copy and an isolated restore drill.

Redis is cache and queue state and is restored empty.

Prometheus and Loki remain retention-bounded tier-2 telemetry unless a later owner decision promotes their history. Alertmanager state is included in monitoring recovery so active silences survive a host loss.

Failure modes

Failure Detection Degraded behavior
OneUptime unavailable External readiness probe and container alerts Alertmanager continues direct Discord/ntfy
PostgreSQL unavailable Readiness and database health checks No incident mutations; direct delivery continues
Redis or workers unavailable Queue and workflow health checks Incidents may persist but workflows pause
ClickHouse unavailable Analytics database health check OneUptime readiness may fail; telemetry remains in Prometheus/Loki
Alertmanager unavailable External probe/dead-man OneUptime cannot compensate for lost routing
Prometheus unavailable External probe/dead-man Existing metric alerts cannot evaluate
Authentik unavailable External SSO probe Human UI is unavailable; machine ingress remains independent
GitHub Actions unavailable Workflow dispatch and queue alerts Investigation continues; remediation is manual
Notification provider failure Delivery canary and workflow status Incident console remains authoritative

The dead-man route must terminate outside the Prometheus/Alertmanager failure domain. OneUptime never becomes the only path for critical delivery.

Upgrade and rollback strategy

OneUptime releases frequently and applies database migrations at startup.

Upgrade:

  1. Review release and security notes.
  2. Test the exact target version against a restored dataset.
  3. Take and verify PostgreSQL and ClickHouse backups.
  4. Pull pinned images and record digests.
  5. Stop ingress and application workers.
  6. Start the new application and allow migrations to finish.
  7. Test readiness, login, incident history, firing/resolved correlation, links, and notification/remediation workflows.

Rollback restores the previous PostgreSQL and ClickHouse backups and the prior exact images together. Reverting an image tag without restoring migrated data is not a rollback.

Implementation sequence

  1. Land this architecture and the alert contract.
  2. Add semantic CI and correct existing rule defects.
  3. Validate and resize infra-services.
  4. Deploy OneUptime as an authenticated shadow stack.
  5. Prove backup and restore before alert ingestion.
  6. Pilot GitHub runner alert correlation.
  7. Roll out alert domains and operational dashboards incrementally.
  8. Connect approval-gated runner remediation.
  9. Migrate lifecycle notifications and add platform self-monitoring.

See operations platform migration for gates and rollback points.

Known risks and deferred work

  • Community Edition requires a second local login behind Authentik because OIDC is an enterprise feature.
  • The five-service incident-only deployment omits OneUptime's probe and AI containers and must be retested on every upgrade.
  • Native workflow find/update behavior must prove duplicate safety under concurrent webhook delivery. A narrow idempotency adapter is allowed only if native workflow primitives fail the acceptance test.
  • The current monitoring host also runs Wazuh and several stateful services. Capacity is a hard gate, not a documentation exercise.
  • Synology, UPS, and external probes need additional exporters before their domains can be represented completely.