Skip to content

Operations platform migration

This runbook moves the homelab from notification-led response to OneUptime incident management without replacing the existing monitoring stack.

The target architecture is documented in operations and incident platform.

Safety rules

  • Prometheus and the Loki ruler remain alert evaluators.
  • Alertmanager remains the routing, inhibition, and silence authority.
  • Existing Discord and ntfy delivery stays active through the shadow and pilot stages.
  • OneUptime does not receive production alerts before its restore drill passes.
  • No remediation action receives an arbitrary command, SSH key, Docker socket, or runner operator credential.
  • Disruptive actions require GitHub protected-environment approval.
  • Stop and roll back a stage if duplicate incidents, lost recovery events, or notification regressions appear.

Stage 0: foundation

Deliver:

  • architecture and state-ownership decision;
  • operational alert contract;
  • Prometheus and Alertmanager semantic CI;
  • corrections for known rule and inhibition defects.

Gate:

uv run python scripts/lint-alert-schema.py
uv run python -m unittest discover -s tests -v
uv run python scripts/lint-markdown.py
promtool check config monitoring/prometheus/prometheus.yml
promtool check rules monitoring/prometheus/alerts/*.yml
promtool test rules monitoring/prometheus/tests/*.yml
amtool check-config /path/to/rendered-alertmanager.yml

Rollback: revert the foundation PR. It does not change live notification routing.

Stage 1: capacity

  1. Measure infra-services CPU, memory, filesystem, and Docker usage.
  2. Measure Proxmox RAM and local-lvm thin-pool headroom.
  3. Agree the target VM allocation and maintenance window with the owner.
  4. Resize VM 123 and verify the guest sees the new allocation.
  5. Update inventory and generated artifacts after the live change.

Gate:

  • at least 20 percent disk free after pulling OneUptime images;
  • at least 6 GiB MemAvailable before OneUptime starts;
  • swap remains enabled;
  • existing stacks are healthy;
  • no InfraServices* capacity alert fires for 15 minutes.

Rollback: CPU and memory can return to the previous allocation after shutdown if required. Disk expansion is not treated as reversible; follow the existing capacity runbook and never improvise a shrink.

Stage 2: shadow deployment

Deploy the five-service core:

  • OneUptime application;
  • OneUptime ingress;
  • PostgreSQL;
  • Redis;
  • ClickHouse.

The probe and AI agent remain disabled. Duplicate telemetry ingestion remains disabled.

Gate:

  • all containers are healthy within the documented startup window;
  • Authentik protects the browser route;
  • OneUptime local authentication is enabled;
  • direct backend access does not bypass authorization;
  • no host ports are published;
  • WebSockets, redirects, cookies, and forwarded HTTPS headers work;
  • the application runs within the approved memory and disk budget for 24 hours.

Rollback: stop the stack and remove its Traefik route. Do not delete volumes until backup/restore testing and owner approval establish whether state is disposable.

Stage 3: data protection

  1. Create a sample incident and workflow run.
  2. Take logical PostgreSQL and consistent ClickHouse backups.
  3. Restore into isolated volumes.
  4. Start the exact pinned OneUptime release against the restored data.
  5. Verify the incident, timeline, workflow history, users, and encrypted variables.
  6. Copy the backup through the tier-1 restic path.

Gate: the isolated restore passes. Production Alertmanager delivery remains blocked until it does.

Rollback: keep the shadow stack stopped and repair the backup procedure.

Stage 4: GitHub runner pilot

Forward only standardized warning and critical GitHub runner alerts.

Alertmanager routing is additive:

  • send_resolved: true;
  • continue: true;
  • existing Discord delivery unchanged;
  • existing critical patching ntfy delivery unchanged.

Synthetic lifecycle:

  1. Send one firing runner alert.
  2. Confirm one OneUptime incident appears.
  3. Repeat the same payload.
  4. Send two concurrent copies.
  5. Confirm the incident count remains one.
  6. Add a second fingerprint to the same correlation.
  7. Resolve one fingerprint and confirm the incident remains open.
  8. Resolve the final fingerprint and confirm the same incident resolves.
  9. Restart OneUptime and verify the timeline remains.

Rollback: remove or disable only the OneUptime child route and reload Alertmanager. Direct receivers continue.

Stage 5: domain rollout

Migrate one alert domain per reviewable change:

  1. node and host availability;
  2. containers and infra-services capacity;
  3. backup and restore;
  4. patching and Komodo;
  5. TLS and service health;
  6. monitoring-platform health.

For each domain:

  • complete alert schema labels and annotations;
  • add rule behavior tests;
  • prove firing, repeat, and resolution;
  • inspect correlation keys;
  • verify direct notifications did not change;
  • add the domain to the operations dashboard.

Rollback: remove the domain matcher from the OneUptime route. Do not remove the Prometheus rule or direct notification route.

Stage 6: runner remediation

The end-to-end example uses the existing runner-fleet dispatch workflow:

  1. A runner-pool alert creates an incident.
  2. The operator acknowledges it in OneUptime.
  3. The operator dispatches exact-runner diagnostics.
  4. GitHub records output as structured artifacts.
  5. The OneUptime timeline receives the workflow-run link and result.
  6. The operator previews drain/remediate/resume.
  7. A protected environment approves disruptive execution.
  8. Prometheus confirms recovery and Alertmanager sends the resolved event.
  9. OneUptime resolves and retains the incident.

Gate:

  • the selected target is not the management runner;
  • Apps, protected environments, and maintenance runner variables are configured;
  • every target exists in infra/github-runners/fleet.yml;
  • actions time out and reject arbitrary commands or unknown targets;
  • early failures still emit an auditable result.

Rollback: disable the OneUptime dispatch credential or workflow action. Existing manual workflow dispatch remains available.

Stage 7: notification cutover

  1. Send OneUptime lifecycle messages to separate pilot Discord and ntfy destinations.
  2. Compare delivery, links, recovery behavior, and noise for at least one week.
  3. Move standardized domain lifecycle messages to the production destinations.
  4. Retain direct Alertmanager delivery for watchdog, OneUptime failure, and other agreed emergency alerts.
  5. Remove permanent duplicate routes only after owner approval.

Rollback: disable OneUptime notification workflows and restore the previous Alertmanager matcher. Incident ingestion can remain active.

Stage 8: closeout

Confirm:

  • operations and platform-health dashboards are provisioned;
  • noisy, stale, frequent, long-running, and flapping alerts are visible;
  • platform health, database health, queue health, webhook delivery, backup freshness, disk, and certificates are monitored;
  • the dead-man path is outside the Prometheus/Alertmanager failure domain;
  • operator, restore, upgrade, rollback, and synthetic-test docs match reality;
  • a project journal entry records the milestone and remaining gaps.

OneUptime becomes canonical incident state only after every prior gate passes.