Skip to content

OneUptime Incident Adapter

Purpose

This runbook deploys and operates the private, race-safe adapter between Alertmanager and OneUptime. The adapter replaces the rejected native workflow for machine correlation. It does not replace OneUptime as the human incident record, and it does not replace direct Discord or ntfy delivery.

The native workflow and its duplicate incidents remain disabled evidence. Do not delete or reuse that workflow.

Production safety state

The GitHub runner, node-availability, container-runtime, infra-services capacity, restic backup-freshness, restore-test health, Komodo configuration, OS patching, and ansible-pull live acceptance gates passed by 2026-08-01. Production routes accept their allowlisted real alerts and the adapter has REQUIRE_SYNTHETIC=false.

These safeguards remain mandatory:

  • continue: true evaluates later sibling routes; it is not receiver-health-based failover and does not guarantee Discord delivery;
  • direct Discord (notify=page) and ntfy (critical service=os-patching) configuration remains unchanged; runner notify=ticket alerts have no direct Discord fallback;
  • the adapter is not exposed through Traefik or a host port;
  • strict alert-name, label, annotation, URL, source-network, and bearer-token validation remains enabled;
  • the rejected native workflow remains disabled.

Architecture

flowchart LR
  P[Prometheus] --> AM[Alertmanager]
  AM -->|"matching page or critical patching route"| D[Discord and ntfy]
  AM -->|"private bearer token"| API[Adapter API]
  API -->|"one transaction"| DB[(Adapter PostgreSQL)]
  DB --> O[Durable outbox]
  O --> W[Adapter worker]
  W -->|"project API key"| OU[OneUptime API]
  W -->|"complete active set"| AM
  PR[Prometheus] -->|"private scrape"| API
  PR -->|"private scrape"| W

PostgreSQL owns delivery durability, event ordering, fingerprint membership, incident generations, retry state, and replay audit. OneUptime is a verified projection. A successful Alertmanager response means PostgreSQL committed; it does not claim OneUptime already converged.

One-time OneUptime setup

These are the only required UI actions. Complete them immediately before the live deployment gate, not while reviewing static code.

Create the two adapter custom fields

The three native-pilot fields already exist:

  • Correlation Key
  • Active Alert Fingerprints
  • Last Seen At

Add two more incident custom fields:

  1. Sign in to https://oneuptime.infra.realemail.app.
  2. Open the homelab operations project.
  3. Open Project Settings.
  4. In the incident settings area, open Custom Fields.
  5. Choose Create Custom Field.
  6. Create Adapter Incident ID:
  7. resource: Incident;
  8. type: Text;
  9. required: No;
  10. description: Stable adapter UUID for remote create adoption.
  11. Create Adapter Revision:
  12. resource: Incident;
  13. type: Text;
  14. required: No;
  15. description: Last adapter projection revision verified remotely.
  16. Confirm all five field names match this runbook exactly, including spaces and capitalization.

Do not repurpose Correlation Key as the remote idempotency key. One correlation can have multiple recurrence generations; Adapter Incident ID is unique to one generation.

Create the project-scoped API key

  1. Open Project Settings → API Keys.
  2. Create a key named incident-adapter.
  3. Give it a documented future expiration. Do not create a non-expiring key.
  4. Open the new key and assign these allow permissions:
  5. CreateProjectIncident
  6. ReadProjectIncident
  7. EditProjectIncident
  8. ReadIncidentState
  9. ReadIncidentSeverity
  10. CreateIncidentStateTimeline
  11. ReadIncidentStateTimeline
  12. ReadIncidentCustomField
  13. Do not grant incident deletion, project administration, or instance-wide permissions.
  14. Reveal and copy the UUID key value into the SOPS edit described below.

The adapter sends ApiKey and ProjectID headers. Do not use the OneUptime master key.

Prepare encrypted secrets

Generate three independent random values:

openssl rand -base64 48 | tr -d '\n/+='
openssl rand -base64 48 | tr -d '\n/+='
openssl rand -base64 48 | tr -d '\n/+='

Use them for:

  • ADAPTER_DATABASE_PASSWORD
  • INCIDENT_ADAPTER_INGEST_TOKEN
  • INCIDENT_ADAPTER_ADMIN_TOKEN

The ingest token must be identical in the adapter and monitoring SOPS files. The database and admin values must not be copied into monitoring.

Edit services/oneuptime-incident-adapter/.env.sops.yaml with:

ADAPTER_DATABASE_PASSWORD: "<generated database password>"
INCIDENT_ADAPTER_INGEST_TOKEN: "<generated ingest token>"
INCIDENT_ADAPTER_ADMIN_TOKEN: "<generated admin token>"
ONEUPTIME_API_KEY: "<project-scoped UUID key>"

Edit services/monitoring/.env.sops.yaml:

  • add INCIDENT_ADAPTER_INGEST_TOKEN with the same ingest token;
  • remove the obsolete ONEUPTIME_ALERTMANAGER_WEBHOOK_PATH entry.

Encrypt both files before committing. Never write the plaintext values to compose.env, shell history, chat, or an issue.

Bootstrap and deploy

The order matters because monitoring declares the ingest network as external.

  1. Merge the reviewed branch.
  2. Pull origin/main on infra-services.
  3. Run host convergence. The common role:
  4. creates oneuptime_adapter_ingest with subnet 172.31.250.0/24;
  5. renders adapter compose.env from the current ansible-pull checkout;
  6. renders monitoring .env;
  7. keeps the public Alertmanager source config mode 0644 so the container's nobody user can read its bind mount.
  8. Deploy the oneuptime-incident-adapter Komodo stack.
  9. Run its post-deploy smoke test.
  10. Redeploy monitoring so Alertmanager and Prometheus join the ingest network.
  11. Reload Prometheus and recreate Alertmanager.

The adapter API readiness depends on PostgreSQL and completed migrations. It does not depend on OneUptime availability; downstream failure must queue rather than reject otherwise-valid Alertmanager deliveries.

Post-deploy smoke test

On infra-services:

cd /opt/homelab/services/oneuptime-incident-adapter
sudo bash scripts/smoke-test.sh --live

The test confirms:

  • PostgreSQL, API, and worker health;
  • at least one applied schema migration;
  • API readiness;
  • worker metrics and heartbeat;
  • API and worker membership on the private ingest network.

Confirm Prometheus sees both adapter targets:

docker exec prometheus wget -qO- \
  'http://localhost:9090/api/v1/query?query=up%7Bjob%3D%22oneuptime-incident-adapter%22%7D'

Confirm Alertmanager retained direct receivers:

docker exec alertmanager amtool config routes show \
  --alertmanager.url=http://localhost:9093

Open the Incident Delivery Control Plane dashboard. It combines Alertmanager notification success/failure with adapter queue safety, worker and reconciliation freshness, validation failures, repairs, and OneUptime API latency. Header links continue into OneUptime, Alertmanager, alert triage, runner state, runner logs, and this runbook.

Synthetic lifecycle acceptance

Keep the native workflow disabled. The acceptance helper talks directly to the adapter API from inside its private container. Its default mode creates a unique synthetic runner correlation:

cd /opt/homelab/services/oneuptime-incident-adapter
sudo bash scripts/acceptance-test.sh --live

For the node and host availability domain, select the focused mode:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=node-availability \
  bash scripts/acceptance-test.sh --live

Container-runtime and infra-services capacity have separate focused modes:

for domain in container-runtime infra-capacity; do
  sudo env ADAPTER_ACCEPTANCE_DOMAIN="${domain}" \
    bash scripts/acceptance-test.sh --live
done

Backup freshness has a focused mode that correlates stale and empty-repository thresholds for one host and backup job:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=backup-freshness \
  bash scripts/acceptance-test.sh --live

Restore health has a focused mode that correlates overdue and failed thresholds for one restore-test host. The production route also admits missing producer metrics into that host-level correlation:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=restore-health \
  bash scripts/acceptance-test.sh --live

Komodo configuration has a focused mode that correlates secret drift and stale detector evidence for the canonical infra-services deployment:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=komodo-config \
  bash scripts/acceptance-test.sh --live

Patching uses one correlated host-health pair plus independent orchestrator and ansible-pull modes:

for domain in patch-host-health patch-orchestrator ansible-pull; do
  sudo env ADAPTER_ACCEPTANCE_DOMAIN="${domain}" \
    bash scripts/acceptance-test.sh --live
done

TLS certificates have a focused mode that correlates one Traefik cn through create, resolve, and recurrence:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=tls-certificate \
  bash scripts/acceptance-test.sh --live

Synthetic acceptance clocks can make Alertmanager endsAt precede generation created_at. The adapter clamps stateChangedAt so OneUptime timeline transitions do not fail with HTTP 400.

Monitoring-platform health has a focused Alloy mode (Loki shares the same service=monitoring route with an independent correlation key):

sudo env ADAPTER_ACCEPTANCE_DOMAIN=monitoring-platform \
  bash scripts/acceptance-test.sh --live

Every mode proves concurrent create, resolution, outbox convergence, verified OneUptime identity, and recurrence. The runner and infra-capacity modes additionally exercise cross-fingerprint correlation and partial resolution; backup-freshness and restore-health exercise warning and critical thresholds, while Komodo exercises multiple warning fingerprints. The runner mode specifically proves:

  1. twenty-five concurrent identical firings commit;
  2. one local incident generation exists;
  3. one semantic event exists for the repeated firing;
  4. a second fingerprint joins the same generation;
  5. partial resolution keeps the generation open;
  6. final resolution closes it;
  7. the outbox reaches succeeded;
  8. OneUptime returns and verifies one incident ID;
  9. a later firing creates recurrence generation two rather than reopening generation one.

In OneUptime, inspect both generated incidents. Confirm:

  • the title ends with one stable [adapter:<uuid>] suffix;
  • Adapter Incident ID matches that suffix;
  • Adapter Revision is populated;
  • Correlation Key is stable across the recurrence;
  • Active Alert Fingerprints is a JSON-encoded text array;
  • Last Seen At is populated;
  • the first incident has an Identified-to-Resolved timeline transition;
  • evidence contains both fingerprints and both resolution events;
  • no duplicate incident exists for either adapter UUID.

Do not delete acceptance incidents. Retain them through the milestone review as evidence.

Current delivery contract

The ordered source tree in monitoring/alertmanager/alertmanager.yml has a blackhole root. Adapter child routes match explicit service, severity and alert-name allowlists, independently of notify. continue: true only continues matching later siblings; it neither retries a failed receiver through Discord nor bypasses Discord's notify=page matcher. The critical OS-patching ntfy route is an earlier sibling with its own matcher. Severity alone does not select Discord.

This table describes receiver selection, not successful delivery. missing means the label is absent. The page runner row is a synthetic routing comparison, not a change to the four production runner rules, which use notify=ticket. "After removal" means removing all adapter child routes, not other receivers.

service alertname severity notify Current receivers After removal
github-runners GitHubRunnerOffline critical ticket oneuptime-incident-adapter blackhole
github-runners GitHubRunnerOffline critical page oneuptime-incident-adapter, discord discord
github-runners GitHubRunnerOffline critical missing oneuptime-incident-adapter blackhole
github-runners UnlistedRunnerAlert critical ticket blackhole blackhole
github-runners UnlistedRunnerAlert critical page discord discord
github-runners GitHubRunnerOffline info ticket blackhole blackhole
os-patching OSPatchOrchestrateFailed critical page ntfy-patching, oneuptime-incident-adapter, discord ntfy-patching, discord
os-patching OSPatchOrchestrateFailed critical ticket ntfy-patching, oneuptime-incident-adapter ntfy-patching

A ticket label alone does not guarantee OneUptime ingestion: both the route and the adapter's stricter payload/domain allowlists must admit it. Missing notify is a schema defect even though an allowlisted alert can select the adapter. Grafana visibility is not notification delivery. OneUptime lifecycle pilot messages use separately configured pilot destinations after projection; they are not an independent fallback for an unavailable adapter or OneUptime.

Alertmanager-path acceptance

First validate the receiver matrix offline. Actual firing/resolved notification checks require a separate owner-approved window and approved destinations; do not send test notifications during a source review. For that later acceptance:

  • record the exact source/rendered revision, labels (including notify), matched receivers, allowlist coverage, silences and inhibition state;
  • for a runner notify=ticket fixture, expect adapter delivery and one OneUptime incident, no Discord; resolution must close the same incident;
  • for a synthetic runner notify=page fixture, expect adapter and Discord; verify each destination separately for firing and recovery without relabeling a real production rule;
  • check critical patching ntfy separately, using its service/severity matcher;
  • verify the rollback matrix offline, then separately approve any live fallback or loss window described in Rollback.

Production routes and REQUIRE_SYNTHETIC=false admit allowlisted real alerts; synthetic="true" is not a production-route isolation gate. Use isolated fixtures for source tests. The child route uses group_by: ["..."]; each webhook contains exactly one fingerprint. The adapter performs correlation across deliveries.

2026-07-31 acceptance evidence

The promotion gate retained the following live evidence. These dated observations are not revalidated here and do not establish today's ticket-to-Discord behavior:

  • direct concurrent lifecycle and recurrence passed after one failed delivery was replayed through the audited endpoint;
  • native Prometheus-to-Alertmanager firing and resolution converged incident 1c9606bc-2d96-44a5-960d-6f4ad7c682a6;
  • Alertmanager notification counters advanced from 9 to 11 for the adapter webhook and from 6 to 7 for Discord;
  • reconciliation repaired two firings missed during compatibility testing and resolved both after complete-set absence;
  • an expired outbox lease was reclaimed after a worker restart;
  • an unreachable Alertmanager recorded ConnectError without mutating local incident state;
  • the final outbox contained seven succeeded rows and no pending, dead, or unknown rows;
  • logical backup 20260731T221056Z passed checksum validation and an isolated restore containing six correlations and seven outbox rows.
  • after promotion, a real-shaped alert without a synthetic label converged OneUptime incident 501d0337-1ea5-4031-b904-51c185c39bfb; both the adapter webhook and Discord counters advanced, resolution reached revision 2, and the outbox contained eight succeeded rows.

The synthetic incidents and failed native-workflow duplicates remain retained evidence. They are not production incidents.

2026-07-31 node-availability evidence

The first Stage 5 domain passed focused acceptance:

  • twenty-five concurrent firings and one exact repeat converged one generation;
  • generation one resolved at revision 27 as OneUptime incident b47239fd-d092-4162-a91a-04fb955a72a0;
  • recurrence created identified generation two as incident 85893b27-f8ff-43ce-a038-feeef4d6fc62;
  • the real firing NodeExporterUnavailable alert for sonarqube converged identified incident 3713a0ff-8384-4d12-9707-ace76975b4c4;
  • Alertmanager webhook and Discord counters both advanced from zero to one for that real delivery, with every failure-reason counter still zero;
  • all three node-domain outbox projections reached succeeded;
  • the Alert Triage dashboard provisioned its Node and host availability panel.

Queue inspection

Read-only queue summary:

docker exec oneuptime-adapter-postgres \
  psql --username incident_adapter --dbname incident_adapter --command '
    SELECT status, count(*), min(created_at) AS oldest
    FROM outbox
    GROUP BY status
    ORDER BY status;
  '

Inspect failures without printing payloads or secrets:

docker exec oneuptime-adapter-postgres \
  psql --username incident_adapter --dbname incident_adapter --command '
    SELECT operation_id, status, attempts, replay_count, updated_at, last_error
    FROM outbox
    WHERE status IN ($$unknown$$, $$dead$$)
    ORDER BY updated_at;
  '

unknown means a create may have committed but its response was lost, so automatic retries stop to prevent duplicates. dead means a permanent failure or repeatedly unexpected adapter exception needs review. Known transient OneUptime HTTP and transport failures retry indefinitely with a bounded five-minute delay; queue-age monitoring reports the downstream outage without turning a recovered outage into manual replay debt.

Audited replay

Before replaying unknown:

  1. Search OneUptime for the exact title and adapter UUID.
  2. If one matching incident exists, leave the row unchanged and investigate why lookup failed.
  3. If multiple incidents exist, do not replay. Preserve evidence and escalate.
  4. Only if no incident exists, authorize replay.

Run the replay from inside the API container so the admin token does not leave the container environment:

docker exec -i oneuptime-incident-adapter python - \
  '<operation-uuid>' '<operator>' '<reason of at least ten characters>' <<'PY'
import json
import os
import sys
import urllib.request

operation_id, actor, reason = sys.argv[1:]
request = urllib.request.Request(
    f"http://127.0.0.1:8080/v1/admin/outbox/{operation_id}/replay",
    data=json.dumps({"actor": actor, "reason": reason}).encode(),
    headers={
        "Authorization": f"Bearer {os.environ['INCIDENT_ADAPTER_ADMIN_TOKEN']}",
        "Content-Type": "application/json",
    },
    method="POST",
)
with urllib.request.urlopen(request, timeout=10) as response:
    print(response.read().decode())
PY

The endpoint accepts only unknown or dead rows and records the previous status, actor, reason, and timestamp in replay_audit. It cannot edit the durable payload.

Reconciliation behavior

Every minute, the worker:

  1. verifies Alertmanager readiness;
  2. fetches the complete active-alert set;
  3. applies the same alert-domain allowlist and strict schema;
  4. repairs remote firings missing locally;
  5. marks local firings missing from Alertmanager;
  6. resolves only after two successful complete scans and ten minutes absent.

An unavailable or malformed Alertmanager response records a failed run and does not close anything.

Backup

Create a logical backup:

cd /opt/homelab/services/oneuptime-incident-adapter
sudo bash scripts/backup.sh

The script writes a PostgreSQL custom-format dump, validates its table of contents with pg_restore --list, writes an adjacent <dump>.sha256 sidecar, and publishes the artifact atomically under backups/. Restic collects that tier-1 directory. The bounded INC-003 source change on fix/restore-artifact-checksums makes new checksum records basename-relative; old sidecars remain unchanged, including absolute original paths.

The backup must contain pending, unknown, and dead outbox rows. Those states are part of the authoritative incident path, not disposable cache.

For recovery, use a full reviewed repository checkout at /opt/homelab, not a copied service directory. The shared checksum helper needs only Python 3's standard library and runs independently of restic staging. Keep the selected dump and its adjacent .sha256 sidecar together. Plan and stage explicit reviewed snapshot IDs using the shared staging contract; its PyYAML/restic prerequisites and PR #469 path-set rules are unchanged.

In a root Bash shell on the Linux recovery host, after reviewing the staged set, verify the selected dump and then run an isolated restore without touching the live volume:

read -r -p "Absolute path to the reviewed staged adapter dump: " ADAPTER_DUMP
python3 /opt/homelab/backups/verify-dump-checksum.py "$ADAPTER_DUMP" || exit 1
bash /opt/homelab/services/oneuptime-incident-adapter/scripts/restore-drill.sh \
  "$ADAPTER_DUMP"

The drill requires the sidecar and calls the helper before any Docker action, including inspection or cleanup. Absence is no longer silently skipped. A valid single unescaped GNU SHA-256 record must name the same filename; the old absolute original path and new relative basename are accepted, but the helper never follows the recorded path. It hashes the exact selected dump and fails on a malformed record, wrong filename, or digest mismatch.

If the sidecar is missing or bad, stop and preserve the evidence. Retrieve the trusted sidecar from the same reviewed backup or an independently authenticated expected digest before preparing a replacement record. Never generate a checksum from the untrusted restored copy just to make the gate pass. See the shared selected-artifact checksum gate.

After verification, the drill still needs Docker, OpenSSL, and its existing PostgreSQL image lookup. It starts a disposable PostgreSQL container on tmpfs, restores with --exit-on-error, checks migration and correlation counts, and removes the container. Record the successful timestamp with the weekly tier-1 restore evidence. A checksum match is not authenticity, freshness, database consistency, or incident-projection correctness; it does not replace those application checks.

This bounded source change does not repair other drill, incident-projection, or restore issues. No production deployment or new database acceptance is claimed. Existing dated backup/restore observations remain historical evidence. Review PR #470 for source validation and the repository README for program status.

Rollback

If adapter ingestion is unhealthy, do not assume removing its route preserves notification coverage. Allowlisted runner tickets become blackhole when their adapter route is removed; only already-matching page/critical-patching routes retain direct destinations. An unavailable adapter does not itself select a new receiver.

Before an owner-authorized rollback:

  1. Enumerate every affected adapter child route and alert class, including active tickets. Record current and proposed receivers using the contract above; retain Prometheus rules and existing direct routes.
  2. For incident-only tickets, obtain owner approval for either a reviewed temporary ticket fallback (destination, exact matchers, expiry and separate delivery acceptance) or an explicit notification-loss window (affected alerts, start/end, named operator, manual triage cadence and escalation path). No fallback is supplied by this runbook. Do not silently relabel tickets as pages or change severity. If neither option is approved, stop.
  3. Review and validate the rendered configuration before deploying it. Remove only the intended adapter child routes; full adapter shutdown requires removing every route targeting it, not just the runner route. Confirm retained direct destinations for their matching labels, not for all alerts.
  4. Preserve the adapter database, outbox, OneUptime incidents and native workflow evidence. Route removal does not cancel durable work or stop worker reconciliation of active alerts. Decide worker/adapter shutdown separately; stop it only after Alertmanager no longer targets it.
  5. At recovery, restore reviewed routes and verify active-set reconciliation, firing/resolved convergence and queue health before ending the loss window or removing a temporary fallback. Alerts that fired and resolved wholly during the gap may be absent from reconciliation; inventory that loss explicitly.

Source tests prove receiver selection only. Provider availability, credentials, inhibition, silences, delivery counters and operator receipt remain separate live acceptance; a successful configuration reload is not that acceptance.

Do not restore the native workflow route. Its concurrent duplicate failure is the reason this adapter exists.

Promotion gate

The 2026-07-31 promotion removed synthetic="true" only after:

  • static CI green, including real-PostgreSQL concurrency tests;
  • direct and Alertmanager-path lifecycle acceptance;
  • concurrent create and concurrent resolution passing;
  • ambiguous-create lookup/adoption passing;
  • audited replay passing;
  • restart and lease recovery passing;
  • reconciliation repair and outage safety passing;
  • logical backup and isolated restore passing;
  • no direct-notification regression;
  • owner review and merge of the promotion PR.

Any future expansion beyond the allowlisted runner, node, container-runtime, infra-services capacity, backup-freshness, restore-health, Komodo configuration, patching, and ansible-pull alerts requires a new reviewed route change and focused lifecycle acceptance.