OneUptime backup and restore¶

OneUptime is tier 1 because it holds canonical incident, acknowledgement, timeline, workflow, and audit state.

The bounded INC-003 checksum change on fix/restore-artifact-checksums is source scope in PR #470, not production deployment or new database-restore acceptance. Use a reviewed revision containing the helper and all three updated writers/drills before following the checksum contract below. The dated observations at the end remain historical evidence, not validation of this change. Follow the repository README and PR #470 for current review and validation status.

Current offsite status: 2026-10-27¶

Both October 27 B2 gaps are closed. Explicitly owner-approved exact-snapshot copies recovered OneUptime 4751b022 as 9fb98052 and incident adapter 5dd10199 as 28878252. Destination originating IDs and tree/time/paths/host matched; see the completed catch-up receipt for full IDs and verification times. The 11:29–11:30 missing-copy observation remains historical evidence, not current copy status.

This closes only those two offsite gaps, not restore integrity, successful retention or future scheduled-run acceptance. Original failed-unit evidence is deliberately untouched; do not rerun a full backup or reset status to make it green. The local-retention fix is merged in #520, but affected live units still lacked --retry-lock=5m at the 11:55 preflight. Normal convergence and next scheduled-run validation remain pending; no forced apply or broader mutation permission follows from the completed two-snapshot approval.

Protected state¶

State	Protection
PostgreSQL	Custom-format `pg_dump`
ClickHouse	Built-in `BACKUP DATABASE` to a dedicated named volume
OneUptime secrets	SOPS-encrypted `.env.sops.yaml` in Git
Exact runtime	Digest-pinned Compose in Git
Redis	Not backed up; cache and queues rebuild empty

ENCRYPTION_SECRET is required with the PostgreSQL restore. A database backup without the matching encrypted SOPS file is not a complete recovery set.

Create a logical backup¶

Run as root on infra-services:

cd /opt/homelab/services/oneuptime
bash scripts/backup.sh

The script:

writes a PostgreSQL custom-format dump to backups/postgres/oneuptime-<timestamp>.dump;
verifies the dump catalog;
runs ClickHouse BACKUP DATABASE oneuptime to oneuptime_clickhouse-backups;
writes an adjacent <dump>.sha256 file with a basename-relative record and a manifest linking both artifacts;
records the live app, ingress, PostgreSQL, Redis, and ClickHouse tag@sha256 image identities plus hashes of compose.yml, config.env, and the encrypted SOPS file.

The backup script derives the OneUptime version from the running app container. It refuses containers that were not started from exact digest-pinned image references. A restore drill likewise refuses an older manifest that lacks the exact datastore images; take a fresh backup instead of guessing a compatible image.

The script also requires the running app and ingress images to equal the checked-out Compose pins. Capture the final source-version recovery set before a target-major checkout reaches /opt/homelab; a mixed source-runtime/target-Git manifest is rejected.

The restic job runs this script as its pre-hook, then backs up the PostgreSQL staging directory and ClickHouse backup volume to the local repository and B2. The producer repair in PR #469 uses one invocation for both roots, tagged homelab-path-set-v1. The tag is a path-set marker, not proof of PostgreSQL/ClickHouse consistency or a valid backup. This is source work, not a claim that the new producer has been deployed or drilled.

The script does not delete old local logical backups. Deleting or relocating backup artifacts requires explicit owner approval. Monitor staging growth and make an owner-approved pruning decision after restic/offsite verification.

Selected-artifact checksum gate¶

This shared gate applies to the PostgreSQL dumps produced by oneuptime, oneuptime-incident-adapter, and oneuptime-remediation-broker. All three writers emit basename-relative SHA-256 records for new dumps. Previously written sidecars remain unchanged, including records with absolute original paths; relocating a recovery set does not require rewriting them.

Use a full reviewed repository checkout at /opt/homelab, not just a copied service directory, and Python 3. backups/verify-dump-checksum.py uses only the Python standard library; it does not require Docker, restic, or PyYAML. This file check runs independently of restic selection/staging, whose dependencies and path-set rules remain unchanged below.

The CLI is python3 /opt/homelab/backups/verify-dump-checksum.py /path/to/reviewed.dump. After staging and reviewing the recovery set, select the actual dump in the Linux recovery host's Bash shell:

read -r -p "Absolute path to the reviewed staged PostgreSQL dump: " POSTGRES_DUMP
python3 /opt/homelab/backups/verify-dump-checksum.py "$POSTGRES_DUMP" || exit 1

The helper requires the adjacent <selected-dump>.sha256 sidecar containing one valid unescaped GNU SHA-256 record for the same filename. It accepts the old absolute original path or the new relative basename, but never follows that recorded path: it hashes the exact selected dump argument. It rejects a missing or malformed sidecar, a different filename, or a digest mismatch. All three drills must pass this gate before any Docker action, including inspection or cleanup; an absent sidecar is no longer silently skipped.

The supported record is 64 hexadecimal digits, a space, a GNU text/binary marker (space or *), and the filename label, with an optional final LF. The record is limited to 64 KiB. Escaped records, backslashes, control characters (including CRLF/NUL), relative directory labels, blank lines, and multiple records are not supported. Dumps and sidecars must be regular files, not final-component symlinks. Their parent directories must be trusted, and neither file may change between verification and import; this is not concurrent-mutation isolation.

On failure, stop and preserve the staged evidence. Retrieve the trusted sidecar from the same reviewed backup, or recover an independently authenticated expected digest before preparing a replacement record. Do not generate a checksum from the untrusted restored copy merely to make the gate pass. Keep the selected dump and sidecar together and rerun verification only after review.

A match establishes byte integrity against the expected digest, not authenticity, freshness, database consistency, or application recoverability. It does not validate the ClickHouse backup, manifest, image identities, secrets, or incident projections. Snapshot completeness and isolated application checks remain separate requirements. This fix does not repair other drill, incident-projection, or restore issues and authorizes no cleanup or live promotion.

Isolated restore drill¶

Run after a successful logical backup, using the reviewed checkout and Python 3 required above. The existing OneUptime drill selects the newest manifest by filename under its backup root; review that manifest and its linked recovery set before running. The checksum fix does not change artifact selection or establish freshness:

cd /opt/homelab/services/oneuptime
bash scripts/restore-drill.sh

The drill:

verifies the exact selected PostgreSQL dump against its mandatory adjacent sidecar using the shared helper, before any Docker action;
restores PostgreSQL into a temporary container and volume;
mounts ClickHouse backup artifacts read-only into a temporary ClickHouse container and volume;
restores the database under oneuptime_restore;
requires non-zero table counts in both datastores;
removes only temporary test containers and volumes.

It does not modify production databases or delete backup artifacts.

Restore from restic¶

Use a root Bash shell on the Linux recovery host, a full reviewed repository checkout at /opt/homelab, Python 3 with PyYAML (python3-yaml), and restic on PATH. Both repository/password-file environment variables are required. See the shared staging contract. /tmp/restore-oneuptime must be absent, with existing non-symlink parents; do not create an empty target or reuse a previous attempt.

umask 077
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/oneuptime
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed complete-path snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/oneuptime/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-oneuptime || exit 1

The default emits metadata/preflight JSON only. Review the selected host, time, ID and both exact roots, then explicitly authorize a staging-only restore:

python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/oneuptime/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-oneuptime --restore || exit 1

For offsite recovery, use this instead of the local attempt, with an absent target. Inspect IDs in B2 rather than assuming local IDs carry over:

umask 077
source /etc/restic/b2.env
read -r -p "Recovery B2 bucket name: " B2_BUCKET
export RESTIC_REPOSITORY="b2:${B2_BUCKET}:oneuptime"
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed complete-path offsite snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/oneuptime/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-oneuptime || exit 1
read -r -p "Type STAGE after reviewing the JSON plan: " STAGING_APPROVAL
test "$STAGING_APPROVAL" = STAGE || exit 1
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/oneuptime/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-oneuptime --restore || exit 1

Older backups remain split, including the dated evidence below. For those, use explicit repeated IDs for reviewed disjoint same-host snapshots covering both manifest roots exactly. Match the logical backup manifest and timestamps yourself; the helper does not prove cross-snapshot consistency. --latest is scheduled verification only and rejects an incomplete newest snapshot without falling back to an older one.

Staging is private (mode 0700); failures preserve partial data. Use a new absent target for retries, updating all references below. The helper does no cutover, promotion or cleanup. Validate both recovered roots, the linked logical backup manifest, checksums, image identities and matching SOPS secrets in isolation before any service stop, deletion, transfer or live copy. Run the selected-artifact checksum gate on the staged PostgreSQL dump, not its original producer path. Root completeness alone is not backup validity. The existing application drill is a separate validation gate; this repair does not certify or fix remaining application-specific recovery defects.

Restic defaults to retention groups by host,paths, so legacy split groups remain separate from new complete-path snapshots. No automatic deletion or grouping change is authorized. Check storage growth; seek separate cleanup approval only after verified local/offsite backups and application restores.

Production recovery¶

Separate owner-approved procedure, not part of the helper contract. Do not execute the retained application steps below until all staging and isolated application validation pass. They are not certified by either the path-set repair or the selected-artifact checksum fix.

Stop ingress and app only after that gate. Leave Redis disposable.
Start empty PostgreSQL and ClickHouse containers at the exact backed-up versions.
Restore the SOPS file and render compose.env; verify the original ENCRYPTION_SECRET is present.
Restore PostgreSQL:

read -r -p "Absolute path to the validated staged PostgreSQL dump: " POSTGRES_DUMP
test -f "$POSTGRES_DUMP" || exit 1
python3 /opt/homelab/backups/verify-dump-checksum.py "$POSTGRES_DUMP" || exit 1
docker exec -i oneuptime-postgres \
  pg_restore --username oneuptime --dbname oneuptime \
  --clean --if-exists < "$POSTGRES_DUMP"

Copy the restored ClickHouse backup set into the oneuptime_clickhouse-backups volume without deleting other sets.
Restore ClickHouse. In the command below, replace REPLACE_WITH_VALIDATED_BACKUP_NAME with the reviewed backup-set name from the matched manifest before execution:

docker exec oneuptime-clickhouse bash -c '
  clickhouse-client \
    --user "$CLICKHOUSE_USER" \
    --password "$CLICKHOUSE_PASSWORD" \
    --query "RESTORE DATABASE oneuptime FROM Disk('\''backups'\'', '\''REPLACE_WITH_VALIDATED_BACKUP_NAME'\'') SETTINGS async = false"
'

Start the app with workflow/remediation integrations disabled.
Verify users, projects, incidents, timelines, workflow runs, and encrypted provider values.
Start ingress and run scripts/smoke-test.sh --live.
Re-enable notification and remediation workflows only after review.

If restoring over an existing ClickHouse database, use an isolated temporary database first. Never issue DROP DATABASE or remove a backup volume without explicit owner approval.

Acceptance¶

PostgreSQL and ClickHouse backups share the same manifest timestamp.
The manifest contains every exact image identity and matching Compose/config/SOPS hashes.
The exact selected PostgreSQL dump passes the mandatory adjacent-sidecar gate.
The isolated drill reports non-zero table counts.
Restic has a local snapshot and tier-1 B2 copy.
The matching SOPS file decrypts on the recovery host.
The exact OneUptime images remain available.
Application readiness and incident history pass after restore.

Last verified¶

On 2026-07-30:

logical backup 20260730T222803Z completed;
isolated restore verified 705 PostgreSQL tables and 42 ClickHouse tables;
the scheduled tier-1 job created local snapshots b25389b3 and 94098a34;
restic copied both paths to B2 as snapshots c0bac798 and 1e81f51b;
OneUptime remained healthy and alert ingestion stayed disabled.