Skip to content

PR 352 — OneUptime v12 platform (owner runbook)

PR: https://github.com/spadoople/homelab/pull/352

This merge pins production OneUptime from 11.7.4 to 12.0.27, adds an isolated Probe and an inert Runner, and keeps deployment on the dedicated Komodo procedure deploy-oneuptime.

Program map: OneUptime v12 merge program. Mechanics: upgrade and rollback, backup and restore, service README.

Already done (PR 351)

You do not redo these:

  • Isolated v12 rehearsal harness is in git on main.
  • 48-hour soak was waived 2026-09-02 after the neighborhood power outage. Do not restart oneuptime-v12-reviewed-soak.service.
  • Custom Probe and Runner identities are already encrypted in services/oneuptime/.env.sops.yaml (PR 352 test plan). Confirm they still exist in the live v11 UI; do not rotate unless login/registration is broken.

Production upgraded to 12.0.27 and passed live acceptance on 2026-09-05. The procedure below is retained for evidence and rollback context.

Progress

# Window You Agent Status
1 Before merge Disable OneUptime notification/remediation workflows; confirm Probe/Runner Backup + restore drill + restic/B2 check; record manifest and SHA Done
2 Merge Approve merge of PR 352 Keep deploy-infra from being used for OneUptime Done
3 After merge, before deploy Confirm UI still v11; do not take another backup Host pull, render compose.env, pull images, policy check Done
4 Deploy Approve production migration Exact approval/pre-deploy; deploy and watch logs Done via documented manual fallback
5 Accept Login, incidents, timelines, and manual lifecycle smoke-test.sh --live; seven containers healthy Done
6 Hold Leave workflows off; leave manual-upgrade in place Observation; do not restore auto-deploy yet Active

1. Before merge (production still v11)

Do this before merging PR 352. backup.sh on the v12 branch refuses a mixed identity (running v11 images vs checked-out v12 Compose). Capture the source-major recovery set while git on the host still matches v11.

1a. Disk (Either)

On infra-services:

df -h /
df -h /var/lib/docker

Need at least 100 GiB free before pulling v12 images. Stop here if not.

1b. Disable OneUptime workflows (You)

oneuptime-incident-adapter is not a workflow. It is a separate Docker stack on infra-services (Alertmanager → adapter API → OneUptime API). You will not see it on this page. Leave it running.

The Workflows table should look like your screenshot:

  • Alertmanager Runner Pilot → Disabled (red). That is the rejected 2026-07-31 native canvas. Do not enable it. Do not delete it.
  • No other notification or remediation workflows enabled.

If that is the only row and it is Disabled, 1b is done. Timelines staying readable is the check.

Do not re-enable that workflow until section 5 says so (it should stay off even then; adapter remains the ingest path).

1c. Confirm Probe, Runner, and provisioner (You)

This is not under WORKFLOWS. Use the left sidebar SETTINGS group (below Workflows / Global Variables / Runs & Logs).

Skip this if it is slowing the backup. Probe/Runner identities are already encrypted in services/oneuptime/.env.sops.yaml from the earlier v11 UI create. Production v11 does not run Probe/Runner containers yet; those start after PR 352 deploys. You are only confirming the dashboard objects still exist so v12 can register.

Look under project Settings for Probes (sometimes under Telemetry / Monitors) and Runners / Copilot / Runbook Agent. Confirm they exist and that Runner execution capabilities are off. If you cannot find the pages, stop and tell the agent — do not create duplicates.

Provisioner API key — optional until after v12 is healthy. Do not block section 1d on this. After cutover:

  1. SETTINGS → API Keys (same project). You should already see incident-adapter. Leave it.
  2. Create a second key named exactly homelab-oneuptime-provisioner only if that name is missing.
  3. Documented expiration (not non-expiring).
  4. Grant only these eight permission keys:
  5. ReadProjectMonitor, CreateProjectMonitor, EditProjectMonitor
  6. ReadMonitorProbe, CreateMonitorProbe, EditMonitorProbe
  7. ReadProjectProbe
  8. ReadProjectMonitorStatus
  9. Do not grant MonitorAdmin, SettingsAdmin, delete permissions, or API-key management.
  10. Store the secret in 1Password. It is not the incident-adapter key.

The provisioner never deletes monitors or Probe assignments. plan / apply wait until section 5.

1d. Logical backup and restore drill (Agent, after 1b)

compose.env is root-owned mode 0600. Running bash scripts/backup.sh as someone fails with rendered environment is not readable. Always use root:

cd /opt/homelab/services/oneuptime
sudo bash scripts/backup.sh
sudo bash scripts/restore-drill.sh

Then confirm restic (paths from backup and restore):

export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/oneuptime
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots

Offsite: B2 copy present for the same job (see the backup runbook).

Record before merge:

Field Value
Git SHA on host (git -C /opt/homelab rev-parse HEAD)
Backup manifest path (backups/oneuptime-<timestamp>.manifest)
oneuptime_version in the manifest (must be 11.7.4)
Restore-drill PostgreSQL table count
Restore-drill ClickHouse table count
Local restic snapshot ID
B2 snapshot ID

If backup.sh says running app/ingress images do not match checked-out Compose, stop. Do not merge 352. The host already has v12 files, or runtime is not the digest-pinned v11 images.

2. Merge PR 352 (You)

Preconditions:

  • Section 1 table is filled.
  • CI on #352 is green (or failures are understood).
  • PR base is main.

Merge the PR. Hosts pull origin/main after merge. Ansible-pull / Komodo ResourceSync will update checkout; they must not start OneUptime via deploy-infra.

3. After merge, before deploy (Agent, you watch)

On infra-services:

  1. Confirm /opt/homelab has the merged SHA.
  2. Confirm OneUptime containers are still oneuptime/app:11.7.4@sha256:… (or the pre-merge digest). docker inspect --format '{{.Config.Image}}' oneuptime-app.
  3. Confirm the Komodo stack oneuptime still has tag manual-upgrade and is absent from deploy-infra.
  4. Do not run backup.sh now. That would be a mixed-identity backup and the script must refuse it.
  5. Render and pull, but do not up:
cd /opt/homelab/services/oneuptime
bash scripts/render-env.sh
docker compose --env-file compose.env config --quiet
docker compose --env-file compose.env pull
python3 scripts/major-upgrade-gate.py check-policy

check-policy must pass with mode = "manual-major-upgrade".

4. Deploy (one-hour window)

The approval file is root-owned mode 0600, binds running v11 images, target v12 images, and the Compose SHA, and expires in 3600 seconds. Create it only when you are ready to click Run.

4a. Write approval (Either, as root)

--actor is your name (the human approving the window), not the agent:

cd /opt/homelab/services/oneuptime
sudo python3 scripts/major-upgrade-gate.py approve \
  --actor "<your name>" \
  --reason "Approved v11 to v12 production maintenance"
sudo python3 scripts/major-upgrade-gate.py pre-deploy

If pre-deploy fails, do not start the procedure. Fix the mismatch; approvals are O_EXCL (one file). A stale/expired file must be handled explicitly — do not overwrite casually.

Komodo periphery has no python3. The oneuptime stack pre_deploy runs bash scripts/komodo-pre-deploy.sh so the same gate executes with host Python. A 2026-09-05 webhook did run deploy-oneuptime; nested DeployStack failed with python3: command not found.

4b. Start only deploy-oneuptime (You)

  1. Open https://komodo.infra.realemail.app.
  2. Procedures → deploy-oneuptime → Run.
  3. Do not run deploy-infra for this. Do not run scripts/trigger-komodo-deploy.py (that procedure is deploy-infra).

Manual equivalent only if Komodo cannot start the stack (agent, same host):

cd /opt/homelab/services/oneuptime
docker compose --env-file compose.env up -d

First boot applies PostgreSQL and ClickHouse migrations. App then ingress readiness can take 9–12 minutes. Watch oneuptime-app and oneuptime-ingress logs. Do not interrupt because the UI is slow in the first few minutes.

5. Accept (You + Agent)

Agent:

cd /opt/homelab/services/oneuptime
bash scripts/smoke-test.sh --live
docker ps --filter name=oneuptime --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'

Expect seven healthy services: app, ingress, postgres, redis, clickhouse, probe, runner. No host ports. Probe and Runner readiness/registration passing. Public URL still redirects through Authentik.

You, in the browser:

  1. Authentik + OneUptime login still work (two layers are intentional).
  2. Projects, existing incidents, and timelines are present.
  3. Encrypted provider values still decrypt (same ENCRYPTION_SECRET).
  4. Drive one synthetic incident lifecycle (open → acknowledge/note → resolve) without turning notification workflows back on.

If this fails, rollback is the tested restore path, not an image-tag downgrade onto an already-migrated database.

6. Hold / observation

  • Keep notification/remediation workflows disabled until you accept the v12 UI.
  • Keep previous v11 images and the section 1 backup through the observation window.
  • Leave the stack tagged manual-upgrade. Restore auto-deploy and put OneUptime back on deploy-infra only after that window, in a follow-up change — not as part of this PR.
  • Desired-state plan / apply waits until v12 is accepted. Use the provisioner key from 1c.

Do not merge PR 353 until this section is green.

Stop / abort rules

  • docker compose --env-file compose.env down stops the runtime and keeps volumes. Never add --volumes without an explicit destroy decision.
  • Rehearsal cleanup is unrelated to production. Do not run upgrade-rehearsal.py stop --confirm as part of this cutover.