Skip to content

PostgreSQL rollout capacity — 2026-10-29

TL;DR

The owner explicitly approved and verified PostgreSQL LXC 102 40 → 80 GiB expansion plus a live retry after the production attempt at 40 GiB was canceled for WAL-retention pressure. Inventory records 80 GiB, with earlier 20/40 GiB evidence preserved below and thin-pool warnings unchanged. Finite live retry reddit-rollout-retry80-20261029 is running, guarded by 8 GB filesystem and 30 GB pool floors; success is not claimed and workers/canary remain unauthorized.

Phase status

Workstream Status
Inventory and docs Verified 80 GiB; disk field only, no CPU/RAM changes
Initial 20 → 40 GiB resize Complete, historical owner evidence below
Isolated rehearsal Schema 16 and preservation counts verified; not production acceptance
Production attempt at 40 GiB Migration 14 manually canceled before 8 GB floor; rollback verified
40 → 80 GiB plus live retry Explicitly owner-approved; resize verified, finite retry running
Proxmox thin-pool capacity Warnings unchanged; monitor guards/alerts, no policy change
Application rollout PR #110 merged; current retry outcome and successful deployment unverified

This advances an in-flight capacity gate, not a completed PLAN phase. PLAN.md is unchanged. The README Owner TODO tracks remaining evidence and acceptance.

Current: 80 GiB verified, finite retry running

The owner supplied successful pct resize 102 rootfs 80G evidence and resize2fs completion at 20,971,520 4K blocks. Post-resize df -B1 /:

Measurement Bytes
Filesystem total 84,426,039,296
Used 15,655,571,456
Available 65,231,896,576

pg_isready reported accepting connections. Pool free space before the 80 GiB resize was 153,167,502 KiB; no post-resize pool reading was supplied. Thin-pool overprovision and absent-autoextend warnings remain; no pool policy changed.

The owner explicitly approved the expansion and live retry to cover observed 35,086,762,440 bytes (~35.09 GB) cumulative WAL, rewrite space, and the 8 GB filesystem floor. The finite retry unit reddit-rollout-retry80-20261029 is now running, with 8 GB filesystem and 30 GB pool guards. Allocation and readiness verification do not prove migration or deployment success. App PR #110 merged as 6da8d1d3c130db1ca27f0300630bc17c479a6840; production acceptance remains pending. No workers or canary are authorized.

Historical: initial 20 → 40 GiB resize

The owner supplied pre-resize pct config 102 rootfs local-lvm:vm-102-disk-0,size=20G, filesystem 21,024,690,176 bytes, and 4.86 GB available (reported units). Inventory then incorrectly recorded 8 GiB.

pct resize 102 rootfs 40G succeeded: LV 20 → 40 GiB, online resize2fs 10,485,760 4K blocks. Post-40-GiB df -B1 / reported 42,158,473,216 bytes total, 15,184,838,656 used, 25,153,007,616 available; pg_isready accepted connections. No precise resize execution time was supplied.

At that earlier capture, Proxmox warned of no thin-pool autoextend protection, 835.01 GiB summed thin volumes exceeding the pool, and 7.89 GiB VG free. Pool available capacity was 174,989,083 KiB before the 40 GiB resize. These are historical readings, not post-80-GiB observations. Pool data/metadata usage, VG free extents, thin virtual allocations and guest headroom are distinct; additional guest capacity does not eliminate pool exhaustion risk.

Historical: isolated retry success

The owner reports the 845,232,040-byte backup restored successfully. After an earlier migration rehearsal timeout, the isolated retry reached schema 16:

  • 1,350,744 session_items preserved.
  • 1,199,224 archive_downloads preserved, with new legacy_evidence preserved.
  • 558,031 archived → legacy_archived transitions were expected.
  • Property counts unchanged: 541 ready, 33 paused, 18 error, 3 cancel.
  • Sampled peak extra space 5,427,712,000 bytes, not a guaranteed upper bound.
  • Total WAL generated 35,086,762,440 bytes; sampled allocation peaked at approximately 1 GB because WAL recycled, not because cumulative generation was small.
  • The original client was interrupted; independent schema/count checks verified success within 60 minutes, not an exact migration duration.

Historical: production cancellation and rollback at 40 GiB

Unlike the rehearsal, active logical replication slot companion_scrape_sub on the other database, tdler, retained WAL throughout the migration transaction. The production cutover was manually canceled during migration 14, before the 8 GB available-space floor. Last sample: 584.2 seconds elapsed, 9,726,587,808 bytes slot lag, 11,545,006,080 bytes available. This sample elapsed time is not a completed migration duration.

Rollback verification: schema 11, no evidence table, zero web DB sessions, Property counts unchanged (541 ready, 33 paused, 18 error, 3 cancel), slot still active with 6,560 bytes lag. New web exited 1, both workers stopped, app offline. App PR #110 was merged but not successfully deployed; the current 80 GiB retry has not yet supplied new completion/health evidence.

Last reported containment: both deploy workflows disabled; Komodo Core/Periphery stopped; Ansible maintenance present and timer disabled; reclaim timer disabled and recovery missing-container error preserved. No unrelated applications stopped. The new finite-retry approval does not imply these controls were lifted or authorize broader automation, workers or canary. Preserve evidence and follow the approved scope.

Source scope and isolation

  • inventory/hosts/postgresql.yaml: initial stale 8 → 40 GiB correction, then 40 → 80 GiB with owner-verification comments; no other inventory values change.
  • Host docs, runbook, and service README distinguish allocation, historical evidence, and pending production acceptance.
  • Existing branch fix/postgresql-rollout-capacity remains isolated from the dirty feat/nocturne-result-filters checkout and app repository. Original-checkout fetch failed on invalid refs/remotes/origin/main 2; the separate bare clone/worktree began at fresh origin/main cfaf1552466ce64e3ccf84276bc84b1f8f2b8b1d. The invalid ref and unrelated work were not repaired, switched or staged.
  • Separate CPU drift (inventory 4 vCPU, earlier observed 6 vCPU) is unchanged. No live operations, scans, app edits, migrations, restarts, backup deletion/relocation, or merge were performed by this documentation agent. Do not shrink the disk or revert inventory to 8/40 GiB to imply live rollback; recovery needs approved scope.

Local validation

Initial checkpoint 9901a97: seven host-inventory generators produced no diffs; 11 CI generator checks plus remediation-target check, schema/cross-references, 53 targeted tests and docs validation passed. Checkpoint 99e2567 used targeted docs/inventory checks; its 27 CI checks subsequently passed. Those are historical results, not validation of a later head or live rollout.

For the 80 GiB update, all seven host-inventory generators ran with no generated diffs; the 11 CI generator checks plus remediation-target check passed. Targeted four-file Markdown lint, host schema, cross-references and structural docs checks passed; semantic comparison confirms only rootfs size changes from 40 to 80. No broad unrelated test suite was rerun; PR #533 records final hook/CI results. Initial npm installation reported five vulnerabilities (two low, three high); no dependency changes are included.

What remains

  • Owner/operator, during the finite retry and before application acceptance: monitor the 8 GB filesystem and 30 GB pool guards, pool data/metadata utilization, VG free extents and actionable alerts. Pool-policy and replication-slot changes require separate approval.
  • Application owner, before production acceptance: capture unit terminal status, schema/preservation counts, space/slot evidence and actual revision/application health. A running or unknown unit is not a pass; no blind retry. Keep workers/canary unauthorized unless separately approved, and do not implicitly release containment.
  • Docs agent, after supplied results: record acceptance or failure accurately without changing app/live state; keep this PR inventory/docs-only and unmerged.
flowchart TD
    A[20 GiB baseline] --> B[40 GiB resize verified]
    B --> C[Isolated schema 16 retry succeeded]
    C --> D[Production WAL retained by cross-database slot]
    D --> E[Migration 14 canceled and rollback verified]
    E --> F[Owner approves 80 GiB and live retry]
    F --> G[80 GiB resize and readiness verified]
    G --> H[Finite live retry running with filesystem and pool guards]
    H --> I[Terminal result and production acceptance pending]