Paperless and LiteLLM logical recovery — 2026-10-24

PR #474 merged REC-004 and REC-005 in source: PostgreSQL 16 logical backup generations and an isolated recovery verifier. This is a source milestone, not a claim of deployment or successful application recovery. The read-only capacity review initially held merge for a bounded storage policy; that follow-up is now implemented in source. The owner merged that source policy; deployment and operational recovery acceptance remain separate, unfulfilled gates in this record.

What changed

• Paperless and LiteLLM backup policies replace hot PGDATA capture with logical generations. Paperless retains its data/media/inbox/export and legacy dump paths; its tier-1 offsite policy is unchanged. LiteLLM remains tier 2, local-only.
• New private generations live at /var/backups/postgres-logical/<service>, root-only and outside the checkout. Each binds a dump to checksum and manifest metadata. The backup-client role provisions root:root mode-0700 directories, installs Python/PyYAML and requires Python 3.11+ with a successful import check before deploying units. Check mode alone does not establish runtime readiness.
• Unlike manual-only PR #473, approved merge and full backup convergence will change future scheduled pre-hooks and path sets. Mounts, repositories, credentials and runtime dependencies must be ready; a systemd-only refresh is not bootstrap.
• The bounded follow-up retains seven newest completed generations, with a 14-slot ceiling, bounded dump/catalog streams and byte/inode headroom checks. It adds guarded post-backup local pruning, not failed-staging or legacy cleanup. Merge/convergence now enables future scheduled deletion under the reviewed policy; the checks are not a filesystem quota or preallocation.

Validation boundary

Initial integrated validation passed 193 + 13 tests. After CI fixture portability corrections, 207 focused local tests passed without skips; the source/fixture CI head passed its 584-test Python suite with two documented environment-related skips. Production safety checks were not weakened. Ruff, YAML/Ansible lint and the strict 213-page documentation build also passed. The bounded-storage follow-up then passed 315 integrated local tests without skips, including actual PromQL evaluation and a real Python 3.10 rejection check. Actual Ansible 2.16.3 condition/template tests also passed; no provisioning role ran. Exact-head results are recorded on the PR. Synthetic fixtures and fake Docker checks are not a real restore: actual PostgreSQL 16 image initialization and restore remain untested.

The recovery verifier is an isolated database smoke test only, not application integrity, media coherence, persistent recovery or production cutover acceptance. No deployment, database queries, backup, live lifecycle operation or cutover was performed in this source-repair slice.

Merge checkpoint and validation follow-up

The owner merged #474 at 52f59c3 on October 25 at 05:13 UTC. The squash tree is identical to the reviewed PR head. Post-merge Lint passed, but Security flagged the same verified source-checksum metadata under the new introducing commit ID. PR #475, merged at 6ca9555, adds only that known squash commit to the existing exact commit/path/digest exception and tests its boundaries. It does not exempt future unrelated commits, broaden paths, rewrite old evidence or change backup behavior. The finding is a source-file digest, not a credential.

No host operations or live cleanup were performed for this follow-up. Merge does not establish effective deployed units, fresh backups or application recovery. The discussed failure-resilience work remains a separately scoped next slice.

Read-only convergence checkpoint

PR #476 records the owner-approved read-only follow-up. The existing scheduled apply completed without intervention. Scoped source hashes, effective backup/apply units and timers, both Python prerequisites, private directory permissions and expected namespace membership match the reviewed configuration.

The overall checkpoint is PARTIAL: monitoring activation is still incomplete. The expected new failure-page rule is absent from loaded rules, despite current source files; a control rule and reload metadata distinguish this from simply having no firing alert. Configuration/rule validation, a bounded targeted reload and exact post-reload rule/health verification need separate approval. No reload was performed.

The service directories had no generation or failure-marker entries at the checkpoint, and earlier successful backup results predate the new files. This is not proof of a new producer/pruner run, NAS/B2 protection, real PG16 restore, notification delivery or application recovery. REC-004/REC-005 remain implemented; 2 deployed, 4 implemented, 247 backlog; none closed.

Raw metadata, timestamps, command/probe hashes, observer-format corrections and validation records remain in the private registry linked from the repository README. No backup behavior or retention policy changed. No forced apply, restart, backup, restore, cleanup or notification ran. Failure-resilience changes remain a separate slice after this operational gap is addressed. The evidence PR also records an unresolved CI failure in an unchanged storage test; assertions and production guards were not weakened to obtain a passing result.

Approved follow-ups — October 25

The owner subsequently approved configuration validation and one conditional Prometheus reload, separately from a synthetic storage-test investigation. Activation stopped at preflight: zero reload attempts and no validator exec. A convergence check unit was in a terminal failed state. A later successful scheduled apply did not erase that failure; no reset, retry, restart, forced apply or guard relaxation was performed. A subsequent read-only observation found another scheduled apply underway, not completed. The check failure needs separate assessment. The earlier absent-rule observation remains historical, not a fresh API check.

The synthetic Linux diagnostic confirmed a content-stability gap: a same-length dump mutation left all compared metadata unchanged, but payload hashes differed. The original assertion still fails; cleanup instrumentation removed only its three cascading fixture errors. This is not evidence of corrupted live backups, but it is a real limitation of metadata-only rechecking. Production storage code is unchanged. The proposed repair must verify original content baselines, preserve partial-deletion/identity guards, and review repeated-hashing I/O cost.

The diagnostic code head ran 787 Linux tests: one failure, no errors, four skips; documentation, monitoring semantic checks and Security passed. Previous failures are preserved. PR #476 remains draft, with exact-head results and private operation/evidence records accessible through the repository README. No lifecycle or recovery-acceptance gate advances.

Content-stability source repair and check investigation

The owner then approved the source repair and a separate read-only investigation. The repair now compares fresh dump size/hash and exact metadata contents with the original validated baselines, independent of timestamps. A full-set content check after authorization precedes mutation; target contents and sibling identities are rechecked through partial deletion. No metadata-keyed hash cache is used.

Content-read amplification is linear: at cap-sized dumps, a 14→7 prune reads/hashes 3.5 GiB for Paperless or 24.5 GiB for LiteLLM, plus bounded metadata/header reads. These are logical-byte projections, not measured disk traffic or runtime. Keepers are not continuously content-checked between deletions, and reads are not atomic snapshots against noncooperating writers. The trusted namespace/shared-lock contract still applies. Retention, artifact format and cleanup authorization are unchanged.

The source head passed 869 Linux tests with four skips. The full local suite, using the existing Bash 5.3 build, passed 873 tests with four skips; 200 native synthetic mutations and the same-tick model were detected. Source-file hash scanner findings were individually verified, with exact commit/path/match exceptions and negative controls rather than a file-wide exemption. Exact final-head results remain on PR #476. REC-004/REC-005 return to implemented, not deployed or closed.

The read-only investigation identified the failed task as Rexit secret parsing. Actual Ansible behavior with a synthetic helper reproduces a skipped check-mode command leaving empty stdout for an unconditional JSON parser. The historical error is protected by no_log, so this is a supported explanation, not uncensored proof of its cause. The previously running scheduled apply completed successfully; that does not erase the failed check. No Rexit role change, credential decryption, unit reset, activation retry or live backup/cleanup occurred.

At that checkpoint, the next decision was how Rexit should handle secret-dependent checks before another activation decision; the storage repair was still unmerged. Subsequent milestones follow below. Prior observations and failed checks are preserved.

Post-merge installed-source checkpoint

The owner merged the checksum follow-up in #477 and the bounded Rexit check-mode fix in #478. Main Lint and Security passed. Three bounded read-only checkpoints then distinguished installed source from successful scheduled exercise.

The final checkpoint on October 25 at 11:44 UTC verified all five allowlisted files against the merged revision: Rexit tasks/defaults, the site playbook, and the two repaired PostgreSQL storage/pruning helpers. No host permissions were changed to obtain those hashes. This is byte-arrival proof, not whole-checkout cleanliness or proof of the exact revision executed by a service.

A successful post-merge check remains unproven. Earlier observations still showed the pre-merge failed check and an active apply. At the final checkpoint the check reported success but had no invocation or start/exit history, while a new apply was active. Those fields cannot establish successful execution. The changed monotonic timing suggests a reboot or timing-context change, but no final boot ID confirmed it. The earlier timer snapshot is historical, not a current deadline.

No fourth checkpoint, forced check/apply, unit reset/restart, decryption, backup, prune, restore or monitoring activation was performed. A later scoped observation must establish fresh invocation context, successful task outcomes, a clean recap and terminal success. Rediscover current monitoring state before considering any reload; the older absent-rule observation is not assumed current after these changes.

Plans, source/probe hashes, all three observations and limitations are retained in the private registry linked from the repository README. Counts remain 2 deployed, 4 implemented, 247 backlog; none closed. Installed bytes, completed execution, monitoring activation and recovery acceptance remain separate.

Fresh boot and schedule context

After #480 merged, a separate read-only snapshot at 12:33 UTC on October 25 observed a stable current boot ID different from the older logged check. It did not establish an exact reboot time or cause. The apply invocation previously seen active had completed with exit status zero; this is service completion, not task or recovery acceptance.

The check still had no recorded current invocation. Its active/waiting timer had a six-hour boot delay, and fresh clock/timer measurements placed its next deadline at about 17:39 UTC (10:39 a.m. PDT). This dated conversion is not a guarantee: clock, suspend, configuration, conditions or lock state can change the result. No older deadline was carried into the new boot context.

Conditional journal reads were skipped because no eligible completed check existed. No source hashes were refreshed, no run was forced, and no further polling was scheduled. The next gate remains a fresh observation after normal scheduling has had time to exercise the repair; other workstreams can proceed independently.

Read-only pre-merge capacity review

The owner-approved follow-up found healthy current storage headroom, not an imminent exhaustion incident. It did not approve indefinite accumulation:

• New snapshots include every completed generation still on disk; restic snapshot retention does not bound this local dump history.
• Retries after downstream failures add generations, while failed staging and catalog scratch need separate budgets.
• Paperless restore staging includes that history again alongside other roots.
• Source-defined byte-pressure and primary-freshness alerts do not establish inode, generation-growth or offsite-freshness protection. The existing run-failure ticket alert also lacks a matching delivery route; live notification delivery was not tested.

That review held merge until a bounded policy was implemented or an explicit finite operating exception approved. No exception, live cleanup or production change was authorized by the review. Existing historical snapshot groups remain preserved. Sanitized measurements, assumptions and capacity scenarios are retained privately through the registry linked from the repository README. Related notification/restore-coverage findings remain open; no duplicate audit roots were created.

Bounded storage follow-up

The owner authorized a source repair, not live deletion. The implemented defaults are:

• Keep seven newest validated generations, not seven calendar days; stop new exports at fourteen. Dump ceilings: 64 MiB Paperless / 512 MiB LiteLLM; producer catalog ceiling: 32 MiB.
• Preserve available bytes of max(15% of total, 8 GiB) + 16 GiB, plus prospective writes; inode floor max(5% of total, 100,000), plus prospective allocation. Preflight precedes Docker/writes and checks recur during streaming. Shared locks coordinate cooperating helpers only; unrelated writers remain outside this guard.
• Only the direct systemd post-backup pruner may delete older eligible generations: exact invocation ID, current control PID, start-post phase and newest matching generation must agree. Required NAS stages—and Paperless B2 stages—must succeed first. Full validation/rechecks precede deletion. Failed/unknown/unsafe entries block work; no forced cleanup or automatic resume is added.
• A narrow critical/page alert covers reported failures for these two jobs through the existing page receiver, including pre-hook failures. Delivery remains untested; the broader ticket-routing finding stays open.

The owner merged this policy, including its scheduled local-deletion behavior. That addresses the unbounded-generation concern in source; it does not prove observed deployment, backup/offsite completeness or successful recovery. No manual live cleanup was performed in this session. Full policy and failure boundaries are in the operating contract and triage guide.

Phase status and next gates

REC-004/REC-005 return to implemented. Progress: 2 deployed, 4 implemented, 247 backlog; none closed.

Concern	Status in this slice	Remaining gate
First two repairs	Previously observed deployed	New recovery sets and isolated application/rollback acceptance
REC-002 / REC-003	Implemented in source	Approved real preparation and application/rollback checks
REC-004 / REC-005	Merged source; backup components observed, overall convergence partial	Complete monitoring activation, then real pipeline/PG16/notification and application/rollback checks; owner operational acceptance
Other 247 groups	Backlog	Separately scoped repairs and evidence

Path-set migration requires separately reviewed historical snapshot manifests for older recovery sets; do not weaken the new manifest to accept old roots. Legacy Paperless dumps require an independently trusted expected digest, not a hash of an untrusted staged copy. Paperless database/media coherence REC-033 remains open. Old raw LiteLLM snapshots are unproven physical backups, not logical recovery evidence; without a trusted logical generation, stop for separate assessment.

Operating guides: Paperless restore, LiteLLM restore, and the private registry linked from the repository README. PLAN.md and the reviewed baseline remain unchanged. Work used an isolated worktree; the shared checkout was left untouched.