Operations platform migration

This runbook moves the homelab from notification-led response to OneUptime incident management without replacing the existing monitoring stack.

The target architecture is documented in operations and incident platform.

Safety rules

• Prometheus and the Loki ruler remain alert evaluators.
• Alertmanager remains the routing, inhibition, and silence authority.
• Existing matching Discord (notify=page) and ntfy (critical service=os-patching) routes stay active through shadow and pilot stages. continue: true evaluates later siblings, not universal failover. Allowlisted runner notify=ticket alerts have no direct Discord fallback.
• OneUptime does not receive production alerts before its restore drill passes.
• No remediation action receives an arbitrary command, SSH key, Docker socket, or runner operator credential.
• Disruptive actions require GitHub protected-environment approval.
• Stop and roll back a stage if duplicate incidents, lost recovery events, or notification regressions appear.

Stage 0: foundation

Deliver:

• architecture and state-ownership decision;
• operational alert contract;
• Prometheus and Alertmanager semantic CI;
• corrections for known rule and inhibition defects.

Gate:

uv run python scripts/lint-alert-schema.py
uv run python -m unittest discover -s tests -v
uv run python scripts/lint-markdown.py
promtool check config monitoring/prometheus/prometheus.yml
promtool check rules monitoring/prometheus/alerts/*.yml
promtool test rules monitoring/prometheus/tests/*.yml
amtool check-config /path/to/rendered-alertmanager.yml

Rollback: revert the foundation PR. It does not change live notification routing.

Stage 1: capacity

1. Measure infra-services CPU, memory, filesystem, and Docker usage.
2. Measure Proxmox RAM and local-lvm thin-pool headroom.
3. Agree the target VM allocation and maintenance window with the owner.
4. Resize VM 123 and verify the guest sees the new allocation.
5. Update inventory and generated artifacts after the live change.

Gate:

• at least 20 percent disk free after pulling OneUptime images;
• at least 6 GiB MemAvailable before OneUptime starts;
• swap remains enabled;
• existing stacks are healthy;
• no InfraServices* capacity alert fires for 15 minutes.

Rollback: CPU and memory can return to the previous allocation after shutdown if required. Disk expansion is not treated as reversible; follow the existing capacity runbook and never improvise a shrink.

Stage 2: shadow deployment

Deploy the five-service core:

• OneUptime application;
• OneUptime ingress;
• PostgreSQL;
• Redis;
• ClickHouse.

The probe and AI agent remain disabled. Duplicate telemetry ingestion remains disabled.

Gate:

• all containers are healthy within the documented startup window;
• Authentik protects the browser route;
• OneUptime local authentication is enabled;
• direct backend access does not bypass authorization;
• no host ports are published;
• WebSockets, redirects, cookies, and forwarded HTTPS headers work;
• the application runs within the approved memory and disk budget for 24 hours.

Rollback: stop the stack and remove its Traefik route. Do not delete volumes until backup/restore testing and owner approval establish whether state is disposable.

Stage 3: data protection

1. Create a sample incident and workflow run.
2. Take logical PostgreSQL and consistent ClickHouse backups.
3. Restore into isolated volumes.
4. Start the exact pinned OneUptime release against the restored data.
5. Verify the incident, timeline, workflow history, users, and encrypted variables.
6. Copy the backup through the tier-1 restic path.

Gate: the isolated restore passes. Production Alertmanager delivery remains blocked until it does.

Rollback: keep the shadow stack stopped and repair the backup procedure.

Stage 4: GitHub runner pilot

The synthetic-only native-canvas evaluation failed its concurrency gate on 2026-07-31. Two simultaneous firing deliveries for one unused correlation key created two incidents. Both workflow runs completed successfully, proving that application-level success does not provide correlation uniqueness.

The workflow is disabled. Do not repeat or continue the native-canvas lifecycle; retained run and incident IDs are in the runner pilot runbook.

Current Alertmanager routing preserves existing direct routes, not universal parallel delivery:

• send_resolved: true requests adapter recovery events;
• continue: true evaluates later sibling matchers;
• synthetic="true" was required during native-canvas evaluation only (adapter promotion removed that requirement for allowlisted domains);
• Discord requires notify=page; runner rules now use notify=ticket;
• critical service=os-patching selects the separate ntfy route;
• adapter service/severity/alert-name allowlists are independent of notify; unmatched alerts reach blackhole.

Native evaluation completed before the stop:

1. One firing alert created one incident.
2. A repeated firing updated that incident and advanced Last Seen At.
3. Two concurrent copies created two incidents.
4. The workflow was disabled and later lifecycle phases were not run.

Stage 4 required a narrow adapter with durable enqueue, unique correlation, atomic fingerprint updates, replay/dead-letter handling, active-alert reconciliation, source restrictions, and rate limiting. That adapter is live: lifecycle gates passed, production routes admit allowlisted domains, and REQUIRE_SYNTHETIC=false. See the incident adapter runbook and the 2026-07-31 / 2026-08-01 journals.

Historical note: during native-canvas evaluation the child route required synthetic="true" and Discord stayed on the direct path. Those dated observations are not current ticket-to-Discord acceptance. Today allowlisted real alerts ingest without a synthetic gate; continued matching does not broaden direct routes.

Rollback must follow the adapter rollback boundary. Removing a runner adapter route leaves its tickets at blackhole. Before removal, approve a reviewed temporary ticket fallback or an explicit owner-approved bounded notification-loss window with named operator and recovery checks. Existing direct receivers remain only for already-matching alerts; do not silently relabel tickets as pages.

Stage 5: domain rollout

Migrate one alert domain per reviewable change:

1. node and host availability;
2. containers and infra-services capacity;
3. backup and restore;
4. patching and Komodo;
5. TLS and service health;
6. monitoring-platform health.

The first domain slice standardizes NodeExporterUnavailable, correlates one incident per inventory-backed instance, and retains direct Discord for its notify=page labels through an additive Alertmanager route. The 2026-07-31 live gate passed concurrent firing, exact repeat, resolution, recurrence, real sonarqube delivery, and unchanged Discord delivery. Re-run it with:

cd /opt/homelab/services/oneuptime-incident-adapter
sudo env ADAPTER_ACCEPTANCE_DOMAIN=node-availability \
  bash scripts/acceptance-test.sh --live

The second slice standardizes container restart/missing telemetry and infra-services root-disk/memory capacity. Container restart rules aggregate to instance,name before alerting so volatile cAdvisor labels cannot enter the incident schema. Infra-services rules select only the canonical job="node" series, avoiding duplicate fingerprints from runner telemetry. After deploy, run both focused gates:

for domain in container-runtime infra-capacity; do
  sudo env ADAPTER_ACCEPTANCE_DOMAIN="${domain}" \
    bash scripts/acceptance-test.sh --live
done

The third slice begins backup and restore with the actionable freshness pair: ResticBackupStale and ResticBackupFailed. Both select only canonical job="node" textfile series, preserve the affected job as backup_service, and correlate warning and critical thresholds by instance,backup_service. Alertmanager inhibition uses the same identity so one failed repo cannot hide stale alerts for unrelated jobs. After deploy, run:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=backup-freshness \
  bash scripts/acceptance-test.sh --live

The fourth slice completes current restore-drill coverage with RestoreTestMetricsMissing, RestoreTestStale, and RestoreTestFailed. Restore drills are host-level and rotate one local tier-1 repository per weekly run, so all conditions correlate by canonical instance rather than the most recently selected repository. The failed metric preserves that repository as optional restore_service context, and the adapter accepts only the five candidates used by the infra-services producer. External tier-1 and tier-2 repositories remain outside this rotation drill. After deploy, run:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=restore-health \
  bash scripts/acceptance-test.sh --live

The fifth slice starts configuration health with KomodoSecretDrift and KomodoSecretDriftCheckStale. Both conditions describe the same canonical infra-services SOPS → compose.env → live-consumer propagation chain, so all secret-specific and detector-health fingerprints share one host-level correlation. Generic Komodo container, backup, restore, and managed-node failures remain in their existing incident domains. After deploy, run:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=komodo-config \
  bash scripts/acceptance-test.sh --live

The patching half of the fifth slice correlates OSPatchStale and PatchRebootPending per inventory-backed host. The infra-services-only OSPatchOrchestrateFailed and per-host AnsiblePullStale retain separate identities because their schedules and remediation paths differ. Freshness is derived from producer success timestamps rather than textfile mtimes, so reboot metric synchronization cannot make an old patch appear current. After deploy, run:

for domain in patch-host-health patch-orchestrator ansible-pull; do
  sudo env ADAPTER_ACCEPTANCE_DOMAIN="${domain}" \
    bash scripts/acceptance-test.sh --live
done

The 2026-08-01 live gate passed all four Komodo and patching lifecycle modes. A separate critical patch route probe reached Discord, ntfy, and the adapter without delivery failures; the resulting OneUptime incident resolved at its second revision. Existing stale ansible-pull and patch-success producers remain real operational work rather than acceptance-generated failures.

The TLS slice standardizes TLSCertExpiringSoon on Traefik's live traefik_tls_certs_not_after metric. Alerts aggregate with min by (instance, cn) so volatile sans/serial labels cannot enter the incident schema, and each certificate common name correlates independently on infra-services. Service-health remains deferred until dedicated rules exist (container runtime already covers restart/missing telemetry). After deploy, run:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=tls-certificate \
  bash scripts/acceptance-test.sh --live

The monitoring-platform slice standardizes CentralAlloyUnavailable and LokiIngestionErrors on live Alloy and Loki scrapes. Loki discards aggregate by instance so volatile reason labels cannot enter the incident schema. AlertmanagerNotificationFailures must sum() the counter increase so volatile integration/reason labels cannot enter the incident schema (otherwise the adapter 400s and Alertmanager counts more webhook clientErrors). PrometheusStoragePressure is retired as a duplicate of migrated infra-capacity disk alerts. Alertmanager, Grafana, and external dead-man self-health remain out of scope until exporters and an outside-failure-domain notifier exist. After deploy, run:

sudo env ADAPTER_ACCEPTANCE_DOMAIN=monitoring-platform \
  bash scripts/acceptance-test.sh --live

For each domain:

• complete alert schema labels and annotations;
• add rule behavior tests;
• prove firing, repeat, and resolution;
• inspect correlation keys;
• verify intended receivers for both notify=page and notify=ticket, including absence of Discord for tickets; source selection is not live delivery acceptance;
• add the domain to the operations dashboard.

Rollback: inventory the domain's tickets before removing its adapter matcher. Follow the adapter rollback boundary, with a reviewed temporary ticket fallback or an owner-approved bounded notification-loss window. Keep Prometheus rules and existing direct routes. Removing the matcher does not cancel queued adapter work or stop worker reconciliation; review shutdown and recovery separately.

Stage 6: runner remediation

The end-to-end path uses the existing runner-fleet dispatcher and a small timeline helper. Do not revive the rejected native OneUptime canvas workflow.

Operator path

1. An allowlisted runner-pool notify=ticket alert enters the adapter and projects a OneUptime incident; it does not select Discord. A page selects Discord only when explicitly labeled notify=page, not because of severity or continue: true.
2. Acknowledge the incident in OneUptime.
3. Open the alert action_url (or runner-fleet-dispatch.yml) and dispatch exact-runner health / drift / report with dry_run: true.
4. Inspect structured artifacts from the run.
5. Record the workflow run on the incident timeline:

# dry-run payload
ONEUPTIME_API_KEY=… ONEUPTIME_PROJECT_ID=… \
  ONEUPTIME_INCIDENT_ID=<uuid> \
  GITHUB_RUN_URL=https://github.com/spadoople/homelab/actions/runs/<id> \
  DRY_RUN=1 \
  python scripts/oneuptime_post_remediation_note.py --conclusion success

Or use workflow oneuptime-remediation-note.yml against the protected oneuptime-remediation environment. 6. Preview drain / remediate / resume with dry_run: true. 7. Apply only after protected-environment approval; never target the management runner. 8. Prometheus recovery + Alertmanager resolved event closes the incident path; OneUptime retains history.

Gate

• the selected target is not the management runner;
• Apps, protected environments, and maintenance runner variables are configured;
• every target exists in infra/github-runners/fleet.yml;
• actions time out and reject arbitrary commands or unknown targets;
• early failures still emit an auditable result;
• runner alerts expose action_url to the dispatcher;
• timeline notes can be posted via scripts/oneuptime_post_remediation_note.py.

Owner setup for live notes

Create GitHub Environment oneuptime-remediation with:

• secrets ONEUPTIME_API_KEY, ONEUPTIME_PROJECT_ID
• optional variable ONEUPTIME_API_BASE (default public https://oneuptime.infra.realemail.app/api)

Rollback: disable the remediation-note workflow or environment secrets. Existing manual workflow dispatch remains available.

Stage 7: notification cutover

Pilot dual-write is implemented in the incident adapter. After a successful OneUptime projection, the worker optionally posts a lifecycle message to pilot-only Discord and ntfy destinations. Production Alertmanager Discord and ntfy routes keep their existing matchers: notify=page for Discord, critical service=os-patching for ntfy. continue: true does not make pilot messages an independent fallback for incident-only tickets; they depend on successful adapter projection. See the receiver matrix.

Enable the pilot

1. Create a dedicated ntfy topic such as homelab-oneuptime-pilot-<4hex> (optional: also a Discord webhook channel — not required; either destination enables pilot notify).
2. Add to services/oneuptime-incident-adapter/.env.sops.yaml:

# PILOT_DISCORD_WEBHOOK_URL: "https://discord.com/api/webhooks/…"  # optional
PILOT_NTFY_SERVER: "https://ntfy.sh"
PILOT_NTFY_TOPIC: "homelab-oneuptime-pilot-…"

1. Re-render and recreate the adapter stack on infra-services:

cd /opt/homelab/services/oneuptime-incident-adapter
bash scripts/render-env.sh
docker compose --env-file compose.env up -d --force-recreate worker

1. Trigger a synthetic or allowlisted lifecycle and confirm messages appear only on the pilot destinations. Production Discord volume must not jump from this path.

Soak checklist (≥1 week)

• [ ] Identified / updated / resolved messages arrive on pilot Discord and ntfy
• [ ] Links open the correct OneUptime incident behind Authentik
• [ ] Noise is acceptable vs direct Alertmanager Discord
• [ ] Recovery (resolve) messaging is timely and not duplicated oddly
• [ ] Adapter outbox success rate stays healthy if a pilot destination fails (pilot notify is best-effort and must not fail projections)

Cutover (owner approval only)

1. Move standardized domain lifecycle messages to production destinations.
2. Retain reviewed direct Alertmanager routes for matching OneUptime failure and other agreed emergency alerts. Keep the independent host heartbeat outside Alertmanager; it is not a monitoring-chain watchdog.
3. Remove permanent duplicate routes only after owner approval.

Rollback: clear PILOT_* secrets, re-render, recreate worker. Incident ingestion can remain active.

Stage 8: closeout

Confirm:

• operations and platform-health dashboards are provisioned;
• noisy, stale, frequent, long-running, and flapping alerts are visible;
• platform health, database health, queue health, webhook delivery, backup freshness, disk, and certificates are monitored;
• the independent host heartbeat path remains outside Prometheus/Alertmanager (monitoring dead-man: host timer → Healthchecks.io → ntfy); its miss drill proves cessation/delivery, not monitoring-plane health;
• monitoring-chain loss detection has separate disposable acceptance; healthy host plus stopped Prometheus/Alertmanager does not stop the current heartbeat;
• complementary Prometheus self-health rules can report partial failures only with a working evaluation and notification path;
• operator, restore, upgrade, rollback, and synthetic-test docs match reality;
• a project journal entry records the milestone and remaining gaps.

OneUptime becomes canonical incident state only after every prior gate passes.