Skip to content

Proxmox storage remediation — proposal

Status: Historical proposal with dated close-out; not an active execution plan Date: 2026-06-24 Host: prox (192.168.6.71) — ASUS PN64, 1 TB NVMe Context: Post–Wave A/B decommission; backup target migrated to Whrrr volume6

This document preserves the June storage audit, original options and recorded outcomes alongside the frozen prox-storage-2026-06-24.md. The 2026-10-25 source-only reconciliation below is not a new storage observation.

Superseded proposal — no operating authorization

LXC 114 nfs-monitoring and 111 influxdb were recorded destroyed on 2026-06-24. Old cleanup, backup, resize and retirement proposals for those instances are closed, not pending work. No reuse of 111 or 114 is established by these records; fresh identity must be verified before any later operation. Owner restrictions remain: saltierpoop stays on prox, Whrrr NFS / Prawns cleanup is out of scope, no backup deletion or relocation without explicit owner approval. Historical commands/options below must not be replayed.

Only the dated residual backlog is retained for future review; every action needs fresh evidence and owner scope approval. No disk, guest, archive or NAS operation was performed for this correction.

Related:

Executive summary (June 2026, before retirement)

The June audit described prox as not in hypervisor-level crisis, with two production-adjacent workloads on different failure modes. This is not a current health assessment; 114 was subsequently retired:

Workload Primary risk Root cause
nfs-monitoring (114) Imminent — guest disk ~full 98.7% thin LV, 91% in-guest, 53 GB Docker (ES stack)
saltierpoop (100) Operational — media I/O Local disk 81% (OK for now); Prawns/Movems NFS at 100%

Three storage layers must be managed independently:

  1. pve-root — OS, ISOs, local scratch (improved; ISO prune still available)
  2. local-lvm thin pool — guest disks (~270 GB thin free; unevenly consumed)
  3. Whrrr NFS — Prawns/synorpn nearly full; infra-backups on volume6 healthy

Recorded decision: ~~Gut or shrink 114~~ 114 retired 2026-06-24. The owner restricted that session to guest disks only, retained infra-backups as the backup target and approved saltierpoop log hygiene. This was not standing approval for future cleanup or NAS work.

Owner decisions (2026-06-24)

Topic Decision
Whrrr NFS / Prawns cleanup Out of scope — pool layout prevents changes; only VM/LXC actions
nfs-monitoring (114) Backup + destroy — executed 2026-06-24
saltierpoop (100) Keep on prox — marginal in-guest cleanup for disk alert relief

Executed (2026-06-24)

Action Result
LXC 114 manual backup infra-backups/dump/manual-lxc-114-rootfs-2026_06_24.tar.zst (~53 GB) + lxc-114-pct.conf
LXC 114 destroy VMID 114 removed; ~110 GB thin pool freed
Prometheus Removed nfs-monitoring-* scrape jobs from prometheus.yml
saltierpoop hygiene journalctl --vacuum-size=500M (~3.4 GB), /tmp clear, syslog.1 truncate → 78% used / ~22% free on /

saltierpoop hygiene suggestion (historical, not standing approval)

Disk alerts fire when root filesystem free space drops below 20% (monitoring/prometheus/alerts/node.yml). Saltierpoop flapped at ~19% free before cleanup.

Original suggested commands, retained as non-replayable history; not a safety certification or scheduling instruction (no Docker prune without review):

# DO NOT RUN: # On saltierpoop (guest agent or SSH)
# DO NOT RUN: sudo journalctl --vacuum-size=500M
# DO NOT RUN: sudo find /tmp -mindepth 1 -mtime +7 -delete
# DO NOT RUN: sudo logrotate -f /etc/logrotate.d/rsyslog   # if syslog grows again

The proposed monthly DSM/Ansible cron was not approved by this record; do not schedule it from this page. Any future hygiene needs fresh measurements and owner approval. The June one-time outcome above remains as recorded.

vzdump + NFS note (historical failure and alternatives)

Large LXC vzdump to infra-backups failed: PVE lxc-usernsexec cannot create .tmp dirs on NFS with the then-recorded Synology squash (Map root to admin only). The original alternatives below are history, not authorization to change NAS permissions or retry a dump of destroyed LXC 114:

  1. DSM NFS rule → Map all users to admin (fixes native vzdump), or
  2. Manual mount + tar | zstd to NFS (used for 114), or
  3. tmpdir on storage pointing to local scratch (only if archive fits ~27 GB free on root)

Resolved (2026-06-24)

No owner decision required — already done.

Item Outcome
Broken CIFS vm-backups on Prawns Removed from PVE
New backup target NFS infra-backups → 192.168.6.215:/volume6/infra-backups (2 TB quota)
Retention keep-last=3 on infra-backups
Surviving Wave A vzdump on prox local 103 unmanic, 117 caddy, 122 netboot last-recorded relocated to infra-backups/dump/; ~3.1 GB total includes logs, not the deleted 105/102/107 dumps
Validation Test vzdump 120 succeeded

Original future-dump example (non-replayable history, not an approved job):

# DO NOT RUN: vzdump <vmid> --storage infra-backups --compress zstd

The proposed 2 TB DSM quota increase was conditional, not executed here or preapproved. Fresh capacity, retention, archive availability and trusted checksum verification remain pending. See backup provenance and recovery gates.

Recorded state (2026-06-24, before later retirements)

Hypervisor layers

Layer Size Used Free Assessment
pve-root (local) 94 GB 63 GB (71%) 27 GB Improved after dump relocation; ~11 GB ISO prune available
local-lvm thin pool 794 GB 537 GB (67.5%) ~270 GB Adequate if 114 stops growing
infra-backups NFS 2 TB quota ~3 GB ~2 TB Healthy
synorpn NFS (Prawns) 30 TB 99.6% ~112 GB Critical upstream cap

Guest thin-pool consumers (top)

VMID Guest Prov. Thin data % ~Actual In-guest / Verdict
100 saltierpoop 260 GB 79.7% ~207 GB 206/258 GB (81%) Largest anchor; local OK, NFS exposed
114 nfs-monitoring 112 GB 98.7% ~110 GB 96/111 GB (91%) Urgent
119 harbor-registry 80 GB 87.8% ~70 GB 45/79 GB (61%) Monitor; registry GC
200 haos 32 GB 95.3% ~30 GB — HA retention review
123 infra-services 30 GB 65.0% ~19 GB — Healthy
111 influxdb 8 GB 86.7% ~7 GB — Retire after Grafana cutover
116 pulse 4 GB 99.5% ~4 GB 59% in-guest Small but maxed LV

The audit recorded 12 guests after Wave A/B (3 VMs, 9 LXCs), before the subsequent retirements. The table and diagram preserve that capture, not a live guest list or fresh storage measurements.

flowchart TB
  subgraph prox["prox NVMe"]
    root["pve-root 27G free"]
    thin["thin pool 270G free"]
    s100["VM 100 · 207G actual"]
    s114["LXC 114 · 110G actual · 98.7%"]
  end
  subgraph whrrr["Whrrr 192.168.6.215"]
    v9["volume9 Prawns · 99.6%"]
    v6["volume6 infra-backups · 0.15%"]
  end
  v9 --> s114
  v9 --> s100
  v6 --> vzdump["vzdump archives"]
  thin --> s100
  thin --> s114

Risk register (historical assessment)

R1 was superseded by 114 retirement; the other likelihoods, sizes and mitigations are June proposals, not newly verified risks or owner approvals. In particular R2 does not override the exclusion of Whrrr cleanup.

ID Risk Likelihood Impact Mitigation owner
R1 114 rootfs fills → ES/Docker crash High Loss of legacy dashboards; possible corrupt indices Owner: keep vs gut 114
R2 Prawns NFS full → saltierpoop *arr/download failures Medium–High Media pipeline stalls Owner: Whrrr archive policy
R3 Large vzdump to local during misconfigured job Low (post-fix) Repeat metrimon failure Ops: always --storage infra-backups
R4 saltierpoop thin growth without cleanup Medium Erodes pool headroom over months Saltbox log/docker hygiene
R5 pulse / haos / harbor thin max Medium Niche service outage Per-guest cleanup or resize
R6 2 TB backup quota exceeded by full VM dump Low until scheduled Backup job failure Raise quota when needed

nfs-monitoring (LXC 114) — superseded proposal

Closed by retirement, not by Option A cleanup

Owner chose backup + destroy, executed 2026-06-24 with a manual rootfs archive after native vzdump failures. Original options below explain the discarded proposal only. Do not clean, resize, back up or start a guest using old ID 114. InfluxDB 111 was also archived and destroyed on 2026-06-24 after the Grafana cutover; the pending language in this original proposal is superseded.

What the June audit recorded

  • Role (original): NFS bridge + observability stack for Prawns/synorpn.
  • Mounts: synorpn (Prawns NFS, 100%), legacy mp1 prawns bind.
  • Running containers: elasticsearch, kibana, logstash, file_crawler, cadvisor, elasticsearch_exporter.
  • Migration status: NFS dashboard metrics moved to Prometheus on infra-services (2026-06-24). Influx on LXC 111 was pending Grafana datasource removal at proposal time; the June journal records subsequent archive and destruction.

Original urgency rationale (superseded by retirement)

  • Thin LV at 98.7% — prox cannot grow the guest much without expansion or cleanup.
  • 53 GB in /var/lib/docker on an 111 GB rootfs with 11 GB free.
  • Upstream Prawns at 100% — file_crawler and any NFS writes are on a full volume.

Options

Option Actions Pros Cons Recommended when
A — Gut legacy stack (default) Stop ES/Kibana/Logstash/file_crawler; keep cadvisor + ES exporter if still scraped; prune Docker volumes Frees 40–50+ GB quickly; reduces ops burden Lose Kibana/Logstash if still used No one opens Kibana weekly
B — Keep stack, shrink data Curate ES indices; Docker prune; reduce retention Preserves UI for debugging Labor-intensive; Prawns still 100% Short-term bridge only
C — Keep stack, expand disk Resize LXC to 150 GB+; set ES ILM Headroom for growth Consumes thin pool; doesn't fix Prawns Confirmed ongoing ES need
D — Full retire Stop LXC; remove from Prometheus scrape; update inventory Maximum prox relief Lose NFS-side tooling entirely Bridge no longer needed

Superseded proposal: “Option A now”, then reconsider D. It was not the final owner choice. Backup + destroy was executed instead, and the nfs-monitoring-* scrape jobs were removed. The old exporter-retention suggestion is not pending.

Owner decisions (114)

D1–D4 closed by supersession, not checked off as executed cleanup: the usage question, Option A approval, B/C sizing choice and file_crawler pause proposal no longer apply to the destroyed instance. Preserve the executed backup/destroy outcome above; do not infer that the unchosen alternatives ran.

saltierpoop (VM 100) — historical alternatives

The owner chose keep on prox plus a marginal in-guest hygiene pass. The old B (NAS cleanup) and C (VMM migration) options below are not approved. Their horizons and measurements are dated proposal context, not a current schedule.

What the June audit recorded

  • Role: Production Saltbox media stack (~40 containers).
  • Prox cost: 30 GB RAM, 260 GB thin disk (~207 GB actual) — ~37% of all thin usage.
  • In-guest disk: 206 / 258 GB (81%) — ~51 GB free on local root.
  • Docker loop: 34 / 54 GB — not the immediate problem.
  • NFS mounts:
Export Volume Use % Free
Prawns volume9 100% ~112 GB
Movems volume2 100% ~87 GB
SerializedWatchables volume1 98% ~938 GB
OrderedWords volume6 43% ~8.3 TB

Why it feels risky (but differently from 114)

  • Prox local disk: Moderate — not imminently full.
  • Media pipeline: High — new writes to Prawns/Movems can fail or stall while exports show 100%.
  • Backup: Full vzdump fits the 2 TB quota; compressed size may be 80–150+ GB. Raise quota before making full-VM dumps routine.

Options

Option Actions Pros Cons Horizon
A — Hygiene only Saltbox log rotation, Docker prune, review / growth Low risk; no architecture change Doesn't fix Prawns Now
B — Whrrr media policy Prawns/Movems cleanup or cold-tier moves Fixes upstream for 100 + 114 Owner media/archive decisions Soon
C — Migrate to Whrrr VMM Move VM off prox per PLAN baseline Frees ~207 GB thin + 30 GB RAM VMM validation, cutover window Phase 9+
D — Resize / split disks Expand LV or add data disk More local room Doesn't fix NFS; uses thin pool Only if local fills

Superseded proposal: “A + B in parallel”, with C as a possible strategic migration from PLAN.md. The recorded owner decision instead excluded Whrrr cleanup and kept saltierpoop on prox. The old proposal cannot override those restrictions.

Owner decisions (100)

  • D5: June risk discussion retained, not a present capacity reading.
  • D6: Whrrr cleanup/tiering excluded by owner; no schedule or approval.
  • D7: One-time log hygiene recorded executed; no standing Docker/image cleanup approval.
  • D8: Keep on prox; no VMM migration authorization.

Secondary guests — original suggestions, not an active backlog

InfluxDB 111 was retired on 2026-06-24; Graylog 109 was recorded operational as central syslog on 2026-06-25, with senders verified 2026-06-26; ISO prune was recorded done 2026-06-25 (~9 GB freed, keeping 24.04 server + alpine). See the June journal. Those rows below are superseded, not remaining retire/revive/prune instructions. Pulse/Harbor/HAOS suggestions require fresh evidence and owner approval; their old percentages do not establish present urgency.

VMID Guest Issue Proposed action Owner gate
111 influxdb Superseded by Prometheus for thermals/NFS Retire LXC after Grafana Influx datasources removed D9: confirm datasources migrated
109 graylog Stopped; Pattern E central syslog Revive per central-syslog-graylog Separate syslog decision
116 pulse 99.5% thin on 4 GB LV Expand to 8 GB or prune in-guest Low urgency
119 harbor-registry 87.8% thin Registry GC, old tag cleanup Low urgency
200 haos 95.3% thin Review HA snapshot/history retention Low urgency
— prox ISOs ~11 GB redundant Ubuntu ISOs Delete desktop + duplicate server ISOs; keep one 24.04 server D10: pick keeper ISO

Whrrr upstream (shared dependency)

Owner 2026-06-24: Whrrr NFS / pool cleanup is out of scope — pool layout prevents changes for now. This proposal covers prox guests only.

114 and 100 both historically depended on volume9 Prawns near capacity. Operational backups use volume6 (infra-backups) — do not route vzdump to Prawns.

Volume Pool Role Use % Proposal
volume9 (Prawns) vg3 Media + synorpn + saltierpoop paths ~100% Owner archive policy; not backup target
volume2 (Movems) vg1 Movies NFS ~100% Same
volume1 (SerializedWatchables) vg1 TV ~98% Monitor
volume6 (infra-backups) vg2 Proxmox vzdump ~0% Keep; raise quota on demand
volume6 (OrderedWords) vg2 Books ~43% Healthy

Capacity alerts: synology-capacity-ntfy (deployed on Whrrr).

Phased execution plan (historical close-out only)

Phase 0 — Done

  • [x] infra-backups NFS storage on prox
  • [x] Retire vm-backups CIFS
  • [x] Relocate surviving legacy vzdump off pve-root (not the deleted artifacts)
  • [x] Document in prox-storage-2026-06-24.md

Phase 1 — Superseded 2026-06-24

The original D1–D4 cleanup plan and its below-70% root / below-85% thin-LV criteria were superseded by backup + destroy of 114, not achieved by the proposed Option A cleanup. Its optional backup instruction is quarantined solely as history:

# DO NOT RUN: vzdump 114 --storage infra-backups

Do not retry that command or resume exporter/file_crawler work against old ID 114.

Phase 2 — Recorded outcomes and exclusions

  • InfluxDB 111: Grafana cutover, archive and retirement recorded 2026-06-24.
  • prox ISO prune: recorded done 2026-06-25 (~9 GB); original ~10 GB was an estimate.
  • saltierpoop: one-time log hygiene recorded, not general Docker prune approval.
  • Whrrr Prawns/Movems capacity actions: out of scope, not a pending phase.
  • Harbor / HAOS / Pulse: no completion inferred; see residual backlog below.

Original targets (pve-root above 35 GB free; Prawns below 98%) were proposal criteria, not verified outcomes or permission to change NAS storage.

Phase 3 — Superseded strategic suggestions

Keep saltierpoop on prox; no VMM migration authorized. Do not reconsider retirement of the already destroyed nfs-monitoring instance. Graylog revive was superseded by the recorded central-syslog outcome, not a remaining storage action.

Owner decision checklist (closed historical questions)

IDs Recorded disposition
D1–D4 Superseded by 114 backup + destroy; not Option A/B/C completion
D5 June capacity-risk discussion, not a new measurement
D6 Whrrr cleanup excluded; no approval
D7 One-time saltierpoop hygiene recorded; repeat work needs approval
D8 Keep saltierpoop on prox; migration not authorized
D9 InfluxDB cutover/archive/retirement recorded 2026-06-24
D10 ISO prune recorded done 2026-06-25; not a new deletion request

No reply to the old D1–D10 form authorizes execution against present guests.

Dated residual backlog (source review 2026-10-25)

Review item Gate before any action
Pulse, Harbor, HAOS capacity/retention suggestions Fresh read-only guest identity, filesystem/thin-pool measurements, workload dependencies and owner-approved scope; do not size from June percentages
saltierpoop repeat hygiene Fresh guest measurements and explicit maintenance approval; keep on prox, no Docker prune by inference
Backup target quota / native vzdump compatibility Separately authorized fresh capacity, permission and recovery review; no quota/squash change or backup run approved here
Archive availability / recovery Exact artifact identity, trusted checksum, isolated restore/integrity/rollback and owner acceptance remain pending; no NAS or alternate-copy search performed

These are review gates, not scheduled jobs. Whrrr NFS / Prawns cleanup remains out of scope; any owner reconsideration is a separate scope decision, not this backlog. Backup deletion or relocation always needs explicit owner approval.

What we are not proposing

  • Deleting backup artifacts without explicit owner approval (see homelab agent rules).
  • Moving vzdump back to Prawns or local as default.
  • Destroying saltierpoop or operating on a guest merely because it has an old VMID; nfs-monitoring is already recorded destroyed.
  • Raising infra-backups quota preemptively (owner: on occasion requiring it).

Changelog

Date Change
2026-10-25 Source-only close-out annotations; stale proposals quarantined, owner restrictions and recovery gates retained; no new live observation
2026-06-24 Initial proposal from live prox audit + owner storage session