OneUptime upgrade and rollback

OneUptime applies PostgreSQL, ClickHouse, and data migrations during startup. Changing an image tag back is not a complete rollback after migrations.

Upgrade prerequisites

• Target release reviewed against official release and upgrade notes.
• Exact app, ingress, PostgreSQL, Redis, and ClickHouse digests recorded.
• Current logical backup and isolated restore drill pass.
• Local restic snapshot and B2 copy are present.
• At least 100 GiB free on infra-services before pulling replacement images.
• Alertmanager direct Discord/ntfy routes remain independent.
• Remediation workflows are disabled during the change.

Staging test

1. Restore the latest PostgreSQL and ClickHouse backups into isolated volumes.
2. Start the target OneUptime images against those volumes.
3. Allow migrations to complete.
4. Verify login, projects, incidents, timelines, runbooks, workflow history, encrypted provider values, and status endpoints.
5. Send a non-production firing/resolved webhook pair.
6. Record migration duration and peak CPU, memory, and disk.

Do not promote a release that cannot read the restored production-shaped data.

Production upgrade

1. Disable OneUptime notification/remediation workflows.
2. Run scripts/backup.sh and scripts/restore-drill.sh.
3. Confirm the tier-1 restic job and B2 copy.
4. Stop OneUptime ingress and app.
5. Update image tags and digests in services/oneuptime/compose.yml.
6. Pull the exact images.
7. Start datastores, then the app.
8. Wait for application readiness; first boot can take 9–12 minutes.
9. Start ingress and run scripts/smoke-test.sh --live.
10. Verify a synthetic incident lifecycle before re-enabling workflows.
11. Retain previous images and pre-upgrade backups through the observation window.

Rollback

Rollback is the tested restore path:

1. Stop ingress and app.
2. Preserve the failed-upgrade databases and logs; do not delete them.
3. Start temporary empty volumes with the previous exact datastore images.
4. Restore the pre-upgrade PostgreSQL and ClickHouse logical backups.
5. Restore the previous app and ingress digests.
6. Render the unchanged SOPS secrets, including the original ENCRYPTION_SECRET.
7. Start app, then ingress.
8. Verify incidents, timelines, workflow history, and readiness.
9. Keep notification/remediation workflows disabled until the operator accepts the rollback.

Do not attempt unsupported down-migrations and do not point an older app image at a database already migrated by a newer release.

Emergency shadow removal

Before production alert ingestion, the shadow runtime can be stopped while retaining state:

cd /opt/homelab/services/oneuptime
docker compose --env-file compose.env down

Do not add --volumes. Any deletion or relocation of database or backup artifacts requires explicit owner approval.