Skip to content

PR 354 — OneUptime remediation broker (owner runbook)

PR: https://github.com/spadoople/homelab/pull/354

PR 353 is merged and accepted with evidence off. Production OneUptime is on v12, and broker mutation remains disabled, so PR 354 merged on 2026-09-05. The private monitoring canary is healthy and scraped by Prometheus. The broker is deployed on infra-services with BROKER_ENABLE_MUTATION=false.

Program map: OneUptime v12 merge program. Canary VM steps also live in GitHub Actions runners once that heading exists on this branch (it ships with 354).

This PR adds a fail-closed typed broker (code + SOPS shape), a private monitoring canary stack, GitHub workflows that dispatch only three named operations, and inventory for Hyper-V guest gha-canary-a. The broker README states the change does not deploy the broker or call GitHub until you enable it.

What merge actually deploys

Piece On merge to main Default
monitoring-canary Compose Yes, auto-deploy on deploy-infra Harmless nginx + metrics; Authentik not involved (no user HTTPS)
Komodo procedure remediation-restart-monitoring-canary-1 ResourceSync Restarts only container monitoring-canary-1
GitHub workflows remediation-*.yml In git enable_mutation comes from the broker env, default false
Broker Compose In git Not a Komodo stack. BROKER_ENABLE_MUTATION=false
gha-canary-a / runner homelab-ephemeral-canary-01 Inventory only You create the VM

Three operations exist, and nothing else:

  • restart_monitoring_canary → canary container restart
  • start_ephemeral_runner → exact-label wake of homelab-ephemeral-canary-01
  • restart_idle_runner → validation-only until GitHub online && !busy + drain/resume exist; do not enable it

Progress

# Task You Agent Status
0 PRs 352–353 accepted Confirm v12 + evidence still off unless you chose section 3 of 353 Confirm canary not required yet Done 2026-09-05
1 Merge PR 354 Review and merge Bring up and verify monitoring-canary Done 2026-09-05
2 Create gha-canary-a Hyper-V clone on CAPTAINKANGAPOO Ansible register exact-label runner Done 2026-09-05
3 Prove exact-label wake Queue a job that needs only homelab-ephemeral-canary-01 Confirm scaler starts only gha-canary-a Done 2026-09-05 (run 34017669466)
4 Secrets (when deploying broker) Fine-grained GitHub token (Actions write, no contents/admin); bearer ≥32 bytes SOPS encrypt; render; keep mutation false Done 2026-09-07. Broker healthy; deny-all 409 in audit.
5 Verifier URLs Supply read-only HTTPS origins + token, or leave unset (deny-all) Confirm deny adapters if any URL is missing
6 24-hour canary Approve mutation only after wake proof Watch kill switch; two failures lock a target Off until you say so

1. Merge

Preconditions:

  • #354 retargeted onto main after 353, CI green.
  • BROKER_ENABLE_MUTATION will remain false in config.env.

Merge. Expect monitoring-canary to appear on infra-services. Confirm Prometheus scrapes it and the failing/healthy rules behave. This is not broker actuation.

2. Create gha-canary-a (You, then Agent)

Owner Hyper-V task on CAPTAINKANGAPOO (Administrator PowerShell). Repo automation cannot create the guest.

Target:

Field Value
VM name gha-canary-a
Address 172.29.240.16
Placement canary-a
Runner homelab-ephemeral-canary-01
Labels runner-remediation-canary, dev, homelab-ephemeral-canary-01 only
Memory Startup 4 GiB, min 2 GiB, max 4 GiB
  1. Confirm .16 is unused from the Hyper-V host.
  2. Clone a general sibling (gha-general-b). Do not clone gha-mgmt (control-plane keys must never land on the canary).
  3. Hostname gha-canary-a, static IP .16, regenerate machine-id and SSH host keys, delete cloned github-runner@* units before joining the runner network.
  4. SSH: ssh runneradmin@172.29.240.16.
  5. Agent: after inventory is on main, run the github-runners play --limit gha-canary-a. It must install only homelab-ephemeral-canary-01.
  6. Confirm GitHub shows that runner with no homelab, ci, deploy, docker, or runner-management labels.
  7. Stop-VM the guest. Queue an exact-label job. Confirm the scaler wakes only gha-canary-a.

Do not enable remediation workflows until wake proof, offline health evidence, and post-start effectiveness all pass.

3. Broker secrets (You + Agent, only when deploying the service)

Copy services/oneuptime-remediation-broker/.env.sops.yaml.example to the real SOPS file. Values:

Key Rule
POSTGRES_PASSWORD Generated; never in git plaintext
BROKER_BEARER_TOKEN ≥ 32 bytes; OneUptime/adapter calls use this
BROKER_GITHUB_TOKEN Fine-grained PAT: Actions write on spadoople/homelab only; no contents or administration write
BROKER_VERIFIER_TOKEN Read-only; only if verifier HTTPS APIs need it

Encrypt for the two age recipients in .sops.yaml. Agent renders compose.env on the host (mode 0600). The broker has no Traefik route and no Authentik — private Compose network only.

Leave BROKER_ENABLE_MUTATION=false. Request bodies cannot flip that flag.

4. Verifiers (fail closed)

Adapters are all-or-nothing. If any read-only setting is missing, runtime stays on deny-only adapters. That is the correct first deploy.

When you are ready to leave deny-all, you supply HTTPS origins without credentials, query strings, or fragments:

  • ONEUPTIME_READ_BASE_URL
  • HEALTH_READ_BASE_URL
  • CHANGE_READ_BASE_URL
  • BACKUP_READ_BASE_URL

Inventory reads committed policy/targets.json (generated from service/fleet YAML). Startup rejects management, database, or storage-shaped names.

Backup freshness: last backup younger than 24 hours and a restore drill within RESTORE_DRILL_MAX_AGE_DAYS (default 90, max 365). Ephemeral runner start is exempt from the backup check. Idle-runner restart stays validation-only until fresh GitHub online && !busy plus drain/restart/resume exist — do not turn that operation on in this PR.

5. 24-hour canary (later, explicit)

Only after:

  • monitoring-canary is healthy;
  • gha-canary-a wake proof passed;
  • broker is running with mutation false and you have seen deny decisions in audit;
  • you explicitly set BROKER_ENABLE_MUTATION=true on the host env (not in a request) and the database kill switch is clear.

Policy still enforces: two accepted attempts per incident, 30-minute target cooldown, ten global actions/hour, two failures lock the target, failed canary sets the global kill switch. Unlock is not in the API.

Do not point the broker at production GitHub runners (homelab-ci-*, homelab-mgmt-01, docker/general CI). Allowlists are monitoring-canary-1, homelab-ephemeral-canary-01, and (later) homelab-ci-01 only for the idle-restart experiment.

Stop / abort

  • Set BROKER_ENABLE_MUTATION=false and recreate the broker container.
  • A tripped kill switch already refuses all accepted actions.
  • docker compose down without --volumes keeps PostgreSQL audit state (tier 1, 365-day retention). Do not delete the volume without an archive decision.
  • Stop-VM gha-canary-a is fine; do not rm the VHDX without asking.