Skip to content

2026-09-05 OneUptime v12 cutover

TL;DR

OneUptime production is live on 12.0.27; all seven services, Authentik edge, and the full live smoke test passed. The pre-upgrade 11.7.4 backup was copied to local restic and B2, then restored successfully with 706 PostgreSQL tables and 42 ClickHouse tables. PR 353 also merged and its LiteLLM/adapter rollout passed. PR 354 merged with its monitoring canary healthy while evidence enrichment, the broker, and remediation mutation remain disabled.

Phase status

  • PR 351 rehearsal: merged; remaining soak owner-waived after the neighborhood power outage.
  • PR 352 v12 platform: merged and accepted live.
  • PR 353 AI evidence: merged; runtime accepted with the bridge off.
  • PR 354 remediation broker: merged; monitoring canary healthy, mutation off.
flowchart LR
  B["Fresh v11 backup<br/>local restic + B2"] --> R["Isolated restore<br/>706 PG / 42 CH tables"]
  R --> M["Merge PR 352"]
  M --> G["Exact-image approval gate"]
  G --> D["Deploy v12.0.27"]
  D --> A["Seven services healthy<br/>browser + smoke accepted"]
  A --> E["PR 353 runtime<br/>evidence remains off"]
  E --> P["PR 354 monitoring canary<br/>healthy; mutation off"]

What shipped

  • PR 352 merged as 1d94ca7.
  • Production application, ingress, Probe, and inert Runner now use exact 12.0.27 image pins.
  • The custom Probe is registered, current, non-global, and restricted to synthetic monitor types.
  • The Runner is registered with runbooks, code-fix tasks, and AI commands disabled.
  • The owner verified login, project history, incident timelines, and a manual create-note-resolve lifecycle.
  • The race-safe incident-adapter API and worker remained healthy; direct Discord/ntfy routing remains independent.
  • PR 353 merged as deba3d1; LiteLLM and the incident adapter were rebuilt and accepted live.
  • The local oneuptime-incident-local-gpt-oss-20b alias is loaded. EVIDENCE_BRIDGE_URL remains empty and no evidence-bridge container exists.
  • PR 354 merged as b8de0a8; Prometheus scrapes monitoring-canary-1, while the broker is not deployed and BROKER_ENABLE_MUTATION=false.
  • gha-canary-a is registered as homelab-ephemeral-canary-01. Exact-label wake proof passed (run 34017669466); the scaler started only that guest.
  • Owner procedures are indexed under the OneUptime v12 merge program.

Known gaps / drift

  • Komodo authenticated the deploy-oneuptime webhook and ran it. Nested DeployStack failed in 8s: periphery pre_deploy python3: command not found. deploy-infra ResourceSync/Procedure webhooks still complete as the git webhook user. HTTP 200 is auth-only; check Updates, not only the OIDC UI.
  • The same HTTP-200-then-look-at-the-wrong-UI pattern recurred after PR 353 and 354; stack-scoped Compose fallbacks still completed those rollouts. deploy-infra Updates from the git webhook user were succeeding. PR 354's first canary start also exposed host-umask drift: mode 0640 bind mounts were unreadable by unprivileged nginx. A live 0644 repair stabilized the canary; the durable image-build fix bakes both files as mode 0444.
  • The migrated Runner record had canRunRunbooks=true even though the container environment requested false. The owner disabled it in the v12 UI; the subsequent registration safety check passed.
  • OneUptime logs report that global SMTP From Email is unset. Notification workflows remain disabled, so this did not block the platform cutover.
  • The stack remains manual-upgrade; do not return it to deploy-infra until the observation window closes in a reviewed follow-up.

What remains

  1. Owner: create homelab-oneuptime-provisioner with the eight documented monitor/Probe permissions if it is still absent; store it in 1Password.
  2. Agent: run desired-state plan, then request owner approval before apply.
  3. Owner: decide whether OneUptime SMTP notifications will be used; if so, configure the global From Email before enabling those workflows.
  4. Agent: Komodo procedure webhooks do execute (async git webhook user). deploy-oneuptime on 2026-09-05 failed in pre_deploy (python3 missing in periphery). Fix: scripts/komodo-pre-deploy.sh.