OneUptime upgrade and rollback

OneUptime applies PostgreSQL, ClickHouse, and data migrations during startup. Changing an image tag back is not a complete rollback after migrations.

Upgrade prerequisites

• Target release reviewed against official release and upgrade notes.
• Exact app, ingress, PostgreSQL, Redis, and ClickHouse digests recorded.
• Current logical backup manifest records those live images plus matching Compose/config/SOPS hashes, and the isolated restore drill passes.
• Local restic snapshot and B2 copy are present.
• At least 100 GiB free on infra-services before pulling replacement images.
• Alertmanager direct Discord/ntfy routes remain independent.
• Remediation workflows are disabled during the change.

Staging test

The reviewed target lives in services/oneuptime/upgrade-rehearsal.env; it must use exact OCI index digests and never a floating tag. Run the rehearsal as root on infra-services:

cd /opt/homelab/services/oneuptime
python3 scripts/upgrade-rehearsal.py start --image-set target
python3 scripts/upgrade-rehearsal.py status
python3 scripts/upgrade-rehearsal.py soak

The harness restores the latest PostgreSQL and ClickHouse backup into isolated temporary volumes, creates an internal-only Docker network, disables queue workers/automatic incidents/alerts/telemetry/LLM/GitHub integration, and publishes no host ports. It records the ingress's internal address and preferred local tunnel port in state.json. It preserves failed resources for investigation rather than cleaning them automatically. Before restoring, it verifies the dump checksum, constrains manifest paths, and requires the current Compose/config/SOPS files to match the backup identity.

During the 48-hour soak:

1. Confirm app and ingress readiness on every 15-minute check.
2. Review results.jsonl under /var/lib/oneuptime-upgrade-rehearsal/ for database table counts and container CPU/memory observations.
3. Read ssh_tunnel_target and ssh_tunnel_local_port from status, then use ssh -L <local-port>:<target> infra-services-cursor from the workstation to verify login, projects, incidents, timelines, runbooks, workflow history, encrypted provider values, and status endpoints.
4. Confirm the production stack remains healthy and receives normal adapter traffic independently.

After the target soak, prove rollback from the untouched v11 backup rather than attempting a down-migration:

python3 scripts/upgrade-rehearsal.py stop --confirm
python3 scripts/upgrade-rehearsal.py start --image-set source
python3 scripts/upgrade-rehearsal.py check
python3 scripts/upgrade-rehearsal.py stop --confirm

stop --confirm removes only the named rehearsal containers, temporary volumes, internal network, and secret runtime environment. It retains the state and results logs and never removes a backup artifact. Cleanup reports failure unless every named Docker resource is gone. Source-version startup is rejected unless the same manifest completed the full continuous target soak and target cleanup.

Do not promote a release that cannot read the restored production-shaped data.

Production upgrade

Owner checklist, merge order, and who clicks what: PR 352 v12 platform in the v12 merge program.

The target-major checkout keeps OneUptime out of deploy-infra. Immediately before the approved maintenance window, verify policy and create a one-hour, root-owned exact-image approval:

cd /opt/homelab/services/oneuptime
python3 scripts/major-upgrade-gate.py check-policy
sudo python3 scripts/major-upgrade-gate.py approve \
  --actor "<operator>" \
  --reason "Approved v11 to v12 production maintenance"
sudo python3 scripts/major-upgrade-gate.py pre-deploy

The approval binds the running source images, configured target images, and Compose SHA-256. Missing, expired, permissive, non-root, or mismatched approval fails closed. Start only the dedicated Komodo deploy-oneuptime Procedure; unrelated eligible stacks use explicit confirmed dispatch of deploy-infra, not an ordinary push.

1. While both runtime and checkout still use the source major, disable OneUptime notification/remediation workflows, run scripts/backup.sh and scripts/restore-drill.sh, and confirm the local restic/B2 copies. Record that manifest and Git SHA before merging target pins.
2. After the target checkout reaches /opt/homelab, confirm OneUptime is still running the source images and remains absent from deploy-infra. Do not create another mixed-identity backup.
3. Render compose.env, validate the Compose model, and pull exact target images.
4. Create and verify the exact approval above, then start only deploy-oneuptime. Komodo periphery has no python3; stack pre_deploy must use bash scripts/komodo-pre-deploy.sh, not a bare python3 invocation.
5. Watch app and ingress migration/readiness logs. Compose keeps datastores healthy before app and app healthy before ingress; first boot can take 9–12 minutes.
6. Run scripts/smoke-test.sh --live.
7. Verify login, projects, incidents, timelines, encrypted provider values, and a synthetic incident lifecycle before re-enabling workflows.
8. Retain previous images and pre-upgrade backups through the observation window.

Rollback

Rollback is the tested restore path:

1. Stop ingress and app.
2. Preserve the failed-upgrade databases and logs; do not delete them.
3. Start temporary empty volumes with the previous exact datastore images.
4. Restore the pre-upgrade PostgreSQL and ClickHouse logical backups.
5. Restore the previous app and ingress digests.
6. Render the unchanged SOPS secrets, including the original ENCRYPTION_SECRET.
7. Start app, then ingress.
8. Verify incidents, timelines, workflow history, and readiness.
9. Keep notification/remediation workflows disabled until the operator accepts the rollback.

Do not attempt unsupported down-migrations and do not point an older app image at a database already migrated by a newer release.

Emergency shadow removal

Before production alert ingestion, the shadow runtime can be stopped while retaining state:

cd /opt/homelab/services/oneuptime
docker compose --env-file compose.env down

Do not add --volumes. Any deletion or relocation of database or backup artifacts requires explicit owner approval.