Skip to content

OneUptime Incident Adapter

Purpose

This runbook deploys and operates the private, race-safe adapter between Alertmanager and OneUptime. The adapter replaces the rejected native workflow for machine correlation. It does not replace OneUptime as the human incident record, and it does not replace direct Discord or ntfy delivery.

The native workflow and its duplicate incidents remain disabled evidence. Do not delete or reuse that workflow.

Production safety state

The GitHub runner, node-availability, container-runtime, infra-services capacity, restic backup-freshness, restore-test health, Komodo configuration, OS patching, and ansible-pull live acceptance gates passed by 2026-08-01. Production routes accept their allowlisted real alerts and the adapter has REQUIRE_SYNTHETIC=false.

These safeguards remain mandatory:

  • continue: true keeps direct Discord delivery active;
  • direct Discord and ntfy configuration remains unchanged;
  • the adapter is not exposed through Traefik or a host port;
  • strict alert-name, label, annotation, URL, source-network, and bearer-token validation remains enabled;
  • the rejected native workflow remains disabled.

Architecture

flowchart LR
  P[Prometheus] --> AM[Alertmanager]
  AM -->|"direct, unchanged"| D[Discord and ntfy]
  AM -->|"private bearer token"| API[Adapter API]
  API -->|"one transaction"| DB[(Adapter PostgreSQL)]
  DB --> O[Durable outbox]
  O --> W[Adapter worker]
  W -->|"project API key"| OU[OneUptime API]
  W -->|"complete active set"| AM
  PR[Prometheus] -->|"private scrape"| API
  PR -->|"private scrape"| W

PostgreSQL owns delivery durability, event ordering, fingerprint membership, incident generations, retry state, and replay audit. OneUptime is a verified projection. A successful Alertmanager response means PostgreSQL committed; it does not claim OneUptime already converged.

One-time OneUptime setup

These are the only required UI actions. Complete them immediately before the live deployment gate, not while reviewing static code.

Create the two adapter custom fields

The three native-pilot fields already exist:

  • Correlation Key
  • Active Alert Fingerprints
  • Last Seen At

Add two more incident custom fields:

  1. Sign in to https://oneuptime.infra.realemail.app.
  2. Open the homelab operations project.
  3. Open Project Settings.
  4. In the incident settings area, open Custom Fields.
  5. Choose Create Custom Field.
  6. Create Adapter Incident ID:
  7. resource: Incident;
  8. type: Text;
  9. required: No;
  10. description: Stable adapter UUID for remote create adoption.
  11. Create Adapter Revision:
  12. resource: Incident;
  13. type: Text;
  14. required: No;
  15. description: Last adapter projection revision verified remotely.
  16. Confirm all five field names match this runbook exactly, including spaces and capitalization.

Do not repurpose Correlation Key as the remote idempotency key. One correlation can have multiple recurrence generations; Adapter Incident ID is unique to one generation.

Create the project-scoped API key

  1. Open Project Settings → API Keys.
  2. Create a key named incident-adapter.
  3. Give it a documented future expiration. Do not create a non-expiring key.
  4. Open the new key and assign these allow permissions:
  5. CreateProjectIncident
  6. ReadProjectIncident
  7. EditProjectIncident
  8. ReadIncidentState
  9. ReadIncidentSeverity
  10. CreateIncidentStateTimeline
  11. ReadIncidentStateTimeline
  12. ReadIncidentCustomField
  13. Do not grant incident deletion, project administration, or instance-wide permissions.
  14. Reveal and copy the UUID key value into the SOPS edit described below.

The adapter sends ApiKey and ProjectID headers. Do not use the OneUptime master key.

Prepare encrypted secrets

Generate three independent random values:

openssl rand -base64 48 | tr -d '\n/+='
openssl rand -base64 48 | tr -d '\n/+='
openssl rand -base64 48 | tr -d '\n/+='

Use them for:

  • ADAPTER_DATABASE_PASSWORD
  • INCIDENT_ADAPTER_INGEST_TOKEN
  • INCIDENT_ADAPTER_ADMIN_TOKEN

The ingest token must be identical in the adapter and monitoring SOPS files. The database and admin values must not be copied into monitoring.

Edit services/oneuptime-incident-adapter/.env.sops.yaml with:

ADAPTER_DATABASE_PASSWORD: "<generated database password>"
INCIDENT_ADAPTER_INGEST_TOKEN: "<generated ingest token>"
INCIDENT_ADAPTER_ADMIN_TOKEN: "<generated admin token>"
ONEUPTIME_API_KEY: "<project-scoped UUID key>"

Edit services/monitoring/.env.sops.yaml:

  • add INCIDENT_ADAPTER_INGEST_TOKEN with the same ingest token;
  • remove the obsolete ONEUPTIME_ALERTMANAGER_WEBHOOK_PATH entry.

Encrypt both files before committing. Never write the plaintext values to compose.env, shell history, chat, or an issue.

Bootstrap and deploy

The order matters because monitoring declares the ingest network as external.

  1. Merge the reviewed branch.
  2. Pull origin/main on infra-services.
  3. Run host convergence. The common role:
  4. creates oneuptime_adapter_ingest with subnet 172.31.250.0/24;
  5. renders adapter compose.env from the current ansible-pull checkout;
  6. renders monitoring .env;
  7. keeps the public Alertmanager source config mode 0644 so the container's nobody user can read its bind mount.
  8. Deploy the oneuptime-incident-adapter Komodo stack.
  9. Run its post-deploy smoke test.
  10. Redeploy monitoring so Alertmanager and Prometheus join the ingest network.
  11. Reload Prometheus and recreate Alertmanager.

The adapter API readiness depends on PostgreSQL and completed migrations. It does not depend on OneUptime availability; downstream failure must queue rather than reject otherwise-valid Alertmanager deliveries.

Post-deploy smoke test

On infra-services:

cd /opt/homelab/services/oneuptime-incident-adapter
sudo bash scripts/smoke-test.sh --live

The test confirms:

  • PostgreSQL, API, and worker health;
  • at least one applied schema migration;
  • API readiness;
  • worker metrics and heartbeat;
  • API and worker membership on the private ingest network.

Confirm Prometheus sees both adapter targets:

docker exec prometheus wget -qO- \
  'http://localhost:9090/api/v1/query?query=up%7Bjob%3D%22oneuptime-incident-adapter%22%7D'

Confirm Alertmanager retained direct receivers:

docker exec alertmanager amtool config routes show \
  --alertmanager.url=http://localhost:9093

Open the Incident Delivery Control Plane dashboard. It combines Alertmanager notification success/failure with adapter queue safety, worker and reconciliation freshness, validation failures, repairs, and OneUptime API latency. Header links continue into OneUptime, Alertmanager, alert triage, runner state, runner logs, and this runbook.

Synthetic lifecycle acceptance

Keep the native workflow disabled. The acceptance helper talks directly to the adapter API from inside its private container. Its default mode creates a unique synthetic runner correlation:

cd /opt/homelab/services/oneuptime-incident-adapter
sudo bash scripts/acceptance-test.sh --live

For the node and host availability domain, select the focused mode:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=node-availability \
  bash scripts/acceptance-test.sh --live

Container-runtime and infra-services capacity have separate focused modes:

for domain in container-runtime infra-capacity; do
  sudo env ADAPTER_ACCEPTANCE_DOMAIN="${domain}" \
    bash scripts/acceptance-test.sh --live
done

Backup freshness has a focused mode that correlates stale and empty-repository thresholds for one host and backup job:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=backup-freshness \
  bash scripts/acceptance-test.sh --live

Restore health has a focused mode that correlates overdue and failed thresholds for one restore-test host. The production route also admits missing producer metrics into that host-level correlation:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=restore-health \
  bash scripts/acceptance-test.sh --live

Komodo configuration has a focused mode that correlates secret drift and stale detector evidence for the canonical infra-services deployment:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=komodo-config \
  bash scripts/acceptance-test.sh --live

Patching uses one correlated host-health pair plus independent orchestrator and ansible-pull modes:

for domain in patch-host-health patch-orchestrator ansible-pull; do
  sudo env ADAPTER_ACCEPTANCE_DOMAIN="${domain}" \
    bash scripts/acceptance-test.sh --live
done

TLS certificates have a focused mode that correlates one Traefik cn through create, resolve, and recurrence:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=tls-certificate \
  bash scripts/acceptance-test.sh --live

Synthetic acceptance clocks can make Alertmanager endsAt precede generation created_at. The adapter clamps stateChangedAt so OneUptime timeline transitions do not fail with HTTP 400.

Monitoring-platform health has a focused Alloy mode (Loki shares the same service=monitoring route with an independent correlation key):

sudo env ADAPTER_ACCEPTANCE_DOMAIN=monitoring-platform \
  bash scripts/acceptance-test.sh --live

Every mode proves concurrent create, resolution, outbox convergence, verified OneUptime identity, and recurrence. The runner and infra-capacity modes additionally exercise cross-fingerprint correlation and partial resolution; backup-freshness and restore-health exercise warning and critical thresholds, while Komodo exercises multiple warning fingerprints. The runner mode specifically proves:

  1. twenty-five concurrent identical firings commit;
  2. one local incident generation exists;
  3. one semantic event exists for the repeated firing;
  4. a second fingerprint joins the same generation;
  5. partial resolution keeps the generation open;
  6. final resolution closes it;
  7. the outbox reaches succeeded;
  8. OneUptime returns and verifies one incident ID;
  9. a later firing creates recurrence generation two rather than reopening generation one.

In OneUptime, inspect both generated incidents. Confirm:

  • the title ends with one stable [adapter:<uuid>] suffix;
  • Adapter Incident ID matches that suffix;
  • Adapter Revision is populated;
  • Correlation Key is stable across the recurrence;
  • Active Alert Fingerprints is a JSON-encoded text array;
  • Last Seen At is populated;
  • the first incident has an Identified-to-Resolved timeline transition;
  • evidence contains both fingerprints and both resolution events;
  • no duplicate incident exists for either adapter UUID.

Do not delete acceptance incidents. Retain them through the milestone review as evidence.

Alertmanager-path acceptance

Before promotion, send a controlled synthetic="true" alert through Prometheus and Alertmanager. Confirm all of the following:

  • Alertmanager reports the adapter notification successful;
  • the adapter delivery count advances;
  • the same alert appears in Discord;
  • one OneUptime incident appears;
  • a resolved notification closes that same incident;
  • no real runner alert enters the adapter route.

The child route uses group_by: ["..."]; each webhook contains exactly one fingerprint. The adapter performs correlation across deliveries.

2026-07-31 acceptance evidence

The promotion gate retained the following live evidence:

  • direct concurrent lifecycle and recurrence passed after one failed delivery was replayed through the audited endpoint;
  • native Prometheus-to-Alertmanager firing and resolution converged incident 1c9606bc-2d96-44a5-960d-6f4ad7c682a6;
  • Alertmanager notification counters advanced from 9 to 11 for the adapter webhook and from 6 to 7 for Discord;
  • reconciliation repaired two firings missed during compatibility testing and resolved both after complete-set absence;
  • an expired outbox lease was reclaimed after a worker restart;
  • an unreachable Alertmanager recorded ConnectError without mutating local incident state;
  • the final outbox contained seven succeeded rows and no pending, dead, or unknown rows;
  • logical backup 20260731T221056Z passed checksum validation and an isolated restore containing six correlations and seven outbox rows.
  • after promotion, a real-shaped alert without a synthetic label converged OneUptime incident 501d0337-1ea5-4031-b904-51c185c39bfb; both the adapter webhook and Discord counters advanced, resolution reached revision 2, and the outbox contained eight succeeded rows.

The synthetic incidents and failed native-workflow duplicates remain retained evidence. They are not production incidents.

2026-07-31 node-availability evidence

The first Stage 5 domain passed focused acceptance:

  • twenty-five concurrent firings and one exact repeat converged one generation;
  • generation one resolved at revision 27 as OneUptime incident b47239fd-d092-4162-a91a-04fb955a72a0;
  • recurrence created identified generation two as incident 85893b27-f8ff-43ce-a038-feeef4d6fc62;
  • the real firing NodeExporterUnavailable alert for sonarqube converged identified incident 3713a0ff-8384-4d12-9707-ace76975b4c4;
  • Alertmanager webhook and Discord counters both advanced from zero to one for that real delivery, with every failure-reason counter still zero;
  • all three node-domain outbox projections reached succeeded;
  • the Alert Triage dashboard provisioned its Node and host availability panel.

Queue inspection

Read-only queue summary:

docker exec oneuptime-adapter-postgres \
  psql --username incident_adapter --dbname incident_adapter --command '
    SELECT status, count(*), min(created_at) AS oldest
    FROM outbox
    GROUP BY status
    ORDER BY status;
  '

Inspect failures without printing payloads or secrets:

docker exec oneuptime-adapter-postgres \
  psql --username incident_adapter --dbname incident_adapter --command '
    SELECT operation_id, status, attempts, replay_count, updated_at, last_error
    FROM outbox
    WHERE status IN ($$unknown$$, $$dead$$)
    ORDER BY updated_at;
  '

unknown means a create may have committed but its response was lost. Automatic retries stop. dead means a permanent or exhausted operation needs review.

Audited replay

Before replaying unknown:

  1. Search OneUptime for the exact title and adapter UUID.
  2. If one matching incident exists, leave the row unchanged and investigate why lookup failed.
  3. If multiple incidents exist, do not replay. Preserve evidence and escalate.
  4. Only if no incident exists, authorize replay.

Run the replay from inside the API container so the admin token does not leave the container environment:

docker exec -i oneuptime-incident-adapter python - \
  '<operation-uuid>' '<operator>' '<reason of at least ten characters>' <<'PY'
import json
import os
import sys
import urllib.request

operation_id, actor, reason = sys.argv[1:]
request = urllib.request.Request(
    f"http://127.0.0.1:8080/v1/admin/outbox/{operation_id}/replay",
    data=json.dumps({"actor": actor, "reason": reason}).encode(),
    headers={
        "Authorization": f"Bearer {os.environ['INCIDENT_ADAPTER_ADMIN_TOKEN']}",
        "Content-Type": "application/json",
    },
    method="POST",
)
with urllib.request.urlopen(request, timeout=10) as response:
    print(response.read().decode())
PY

The endpoint accepts only unknown or dead rows and records the previous status, actor, reason, and timestamp in replay_audit. It cannot edit the durable payload.

Reconciliation behavior

Every minute, the worker:

  1. verifies Alertmanager readiness;
  2. fetches the complete active-alert set;
  3. applies the same alert-domain allowlist and strict schema;
  4. repairs remote firings missing locally;
  5. marks local firings missing from Alertmanager;
  6. resolves only after two successful complete scans and ten minutes absent.

An unavailable or malformed Alertmanager response records a failed run and does not close anything.

Backup

Create a logical backup:

cd /opt/homelab/services/oneuptime-incident-adapter
sudo bash scripts/backup.sh

The script writes a PostgreSQL custom-format dump, validates its table of contents with pg_restore --list, writes SHA-256 metadata, and publishes the artifact atomically under backups/. Restic collects that tier-1 directory.

The backup must contain pending, unknown, and dead outbox rows. Those states are part of the authoritative incident path, not disposable cache.

Run an isolated restore without touching the live volume:

sudo bash scripts/restore-drill.sh \
  backups/postgres/oneuptime-incident-adapter-<timestamp>.dump

The drill verifies the checksum when present, starts a disposable PostgreSQL container on tmpfs, restores with --exit-on-error, checks migration and correlation counts, and removes the container. Record the successful timestamp with the weekly tier-1 restore evidence.

Rollback

If adapter ingestion is unhealthy:

  1. Leave direct Discord and ntfy delivery active.
  2. Remove or disable only the oneuptime-incident-adapter child route.
  3. Recreate Alertmanager and confirm direct delivery.
  4. Do not delete the adapter database, outbox, OneUptime incidents, or native workflow evidence.
  5. Stop the adapter only after Alertmanager no longer targets it.

Do not restore the native workflow route. Its concurrent duplicate failure is the reason this adapter exists.

Promotion gate

The 2026-07-31 promotion removed synthetic="true" only after:

  • static CI green, including real-PostgreSQL concurrency tests;
  • direct and Alertmanager-path lifecycle acceptance;
  • concurrent create and concurrent resolution passing;
  • ambiguous-create lookup/adoption passing;
  • audited replay passing;
  • restart and lease recovery passing;
  • reconciliation repair and outage safety passing;
  • logical backup and isolated restore passing;
  • no direct-notification regression;
  • owner review and merge of the promotion PR.

Any future expansion beyond the allowlisted runner, node, container-runtime, infra-services capacity, backup-freshness, restore-health, Komodo configuration, patching, and ansible-pull alerts requires a new reviewed route change and focused lifecycle acceptance.