infra-services capacity and resize¶

Capacity targets, operating guidelines, and Proxmox resize procedure for infra-services (Proxmox VM 123, 192.168.6.17).

Inventory target (inventory/hosts/infra-services.yaml): 12 vCPU / 28 GiB RAM / 200 GiB disk (Proxmox), 190 GiB ext4 in guest, 4 GiB swap. Resized 2026-07-29 for the OneUptime incident-platform deployment after correcting inventory drift from an intermediate 8 vCPU / 22 GiB / 135 GiB allocation.

Prometheus rules scoped to this host live in monitoring/prometheus/alerts/infra-services.yml.

Recorded observation (2026-10-24)¶

Read-only qm config 123 reports 8 vCPU, 28 GiB RAM and 200 GiB scsi0. Inventory still specifies 12 vCPU. Reconcile that discrepancy before treating inventory as the live allocation; this documentation audit did not resize the VM. This is a dated observation, not a fresh live check. The following sizing and utilization tables are the July rollout record. Wazuh is now retired; do not re-enable its permission-repair hooks.

Target sizing¶

Resource	OneUptime pilot floor	July rollout	Notes
RAM	24 GiB	28 GiB	Existing stacks use ~8–9 GiB; room for the five-service OneUptime core
Disk	180 GiB	200 GiB	~115 GiB free after resize; preserves image, database, and upgrade headroom
Swap	4 GiB	4 GiB swapfile	Safety net only — not a substitute for RAM
vCPU	8	12	OneUptime adds PostgreSQL, Redis, ClickHouse, and workers

Current OneUptime gate: do not deploy if normal-state MemAvailable falls below 12 GiB or / has less than 100 GiB free before the OneUptime images and databases are created.

Update inventory after the Proxmox resize (not before):

hardware:
  vcpu: 12
  ram_gb: 28
  storage:
    - device: scsi0
      size_gb: 200
      kind: virtio-scsi
      role: boot+data

Then run all inventory generators and commit.

Measured baseline (2026-07-29, pre-OneUptime)¶

Signal	Value
Guest allocation	12 vCPU, 28 GiB RAM, 200 GiB disk
Guest `/`	190 GiB total, 75 GiB used, 115 GiB free
Guest memory	~8.4 GiB used, ~18 GiB available after startup
Guest swap	4 GiB, unused after restart
Containers	28 running; all health checks green after recovery
Proxmox memory	~17 GiB available after VM restart
`local-lvm`	~321 GiB free after disk expansion
Largest RSS	LiteLLM ~1.9 GiB, Wazuh indexer ~1.5 GiB, Stirling ~1.2 GiB

The first graceful reboot attempt hung after networking and the guest agent stopped. VM 123 required a QEMU stop/start. Wazuh then failed because Komodo's checkout ownership had made its mode-0400 TLS keys unreadable by UID 1000. Running services/wazuh/scripts/fix-cert-permissions.sh restored all Wazuh components. That repair was required at the time. Wazuh has since been retired; its repair hooks must remain disabled.

Measured baseline (2026-06, post-Wazuh)¶

Bucket	Size
Docker images	~13 GiB (Wazuh ~7 GiB)
Container layers + volumes	~6 GiB
OS + `/usr`	~3 GiB
Root `/` total	~28 GiB on 29 GiB VM
Container RSS (steady)	~3.2–3.8 GiB
Host overhead	~0.8–1.2 GiB

Largest data volumes at snapshot: Prometheus ~1.5 GiB, AdGuard ~558 MiB, Mongo ~507 MiB, Loki ~320 MiB. Wazuh indexer data was still tiny at that snapshot. The stack has since been removed.

Safe operating guidelines¶

Disk¶

Never run below 15% free on / — treat 85% used as warning, 90% as incident (pulls and indexer merges fail badly when full).
After a large stack upgrade: within 24h run docker image prune -f once the new stack is healthy. Budget headroom during pull if old images are still present.
Tight disk upgrade pattern: prune old images first, or pull one service at a time and prune between.
Monitoring is bounded (Prometheus + Loki 30d retention) — budget ~6 GiB combined at steady state. Reserve another 50–100 GiB for OneUptime images, PostgreSQL, ClickHouse, upgrade overlap, and local backup staging.

Memory¶

Do not add another heavy stack without revisiting this budget.
Enable the 4 GiB swapfile (Ansible) before or with the RAM resize — cheap insurance during pulls and patch windows.

Wazuh-specific¶

The stack was removed. Do not size the host for a Wazuh indexer, and do not reinstall agents. Graylog remains the log path.

Emergency disk relief (before resize)¶

When / is critical and you need headroom immediately:

# On infra-services (SSH as someone)
docker image prune -f
docker builder prune -f   # if build cache exists
sudo journalctl --vacuum-size=200M
df -h /

Prune old image tags only after the replacement stack is verified healthy.

Future resize procedure¶

Grow-only guidance, not shrink authorization. Before any change, re-establish VM identity, measure the disk/partition/filesystem boundaries and free space, and obtain owner approval plus a verified recovery and rollback plan. Historical sizes below are not targets. A snapshot alone is not an independent backup. Any future shrink requires a separately approved design and disposable offline-clone validation; this runbook supplies no replacement shrink recipe.

Capture guest CPU, memory, filesystem, swap, Docker RSS, and image/volume usage.
Capture Proxmox memory and local-lvm thin-pool headroom.
Get owner approval for the exact CPU, memory, and disk target.
Expand scsi0 only; Proxmox cannot shrink it safely.
Grow partition 1, then use an explicit resize2fs target larger than the measured filesystem and within the verified partition boundary. Stop if those grow-only bounds cannot be established; do not reuse historical constants.
Apply CPU/RAM with a full VM stop/start; an in-guest reboot does not recreate QEMU with the new allocation.
Wait for every existing container health check, then verify at least 20 percent disk free and 12 GiB MemAvailable.
Update inventory after reality changes and run all generators.

Historical 2026-06 resize procedure¶

Historical record — not a maintenance procedure

The following 100 GiB / 16 GiB instructions record the June recovery from the accidental 34 TiB disk. Do not replay these instructions or reuse their values. They are superseded by the July close-out and the dated observation above. For future work use the grow-only guidance.

The original June plan put disk before RAM, with swap before or after RAM. Its instructions below are retained as historical context, not a scheduled task.

1. Pre-checks¶

# From your workstation
ssh infra-services-cursor 'df -h /; free -h; docker ps --format "{{.Names}}" | wc -l'

On prox (192.168.6.71): confirm local-lvm (or the VM’s datastore) has enough thin free space for +50 GiB. See prox storage remediation.

Optional: snapshot VM 123 in Proxmox before resize.

2. Extend disk (Proxmox)¶

Use GiB, not TiB

Proxmox Resize disk asks for the new total size. Enter 100 and confirm the unit is GiB. A typo of 34000 GiB (or selecting TiB) produced a ~34 TiB virtio disk in the 2026-06-17 incident — the guest partition grew to match while ext4 stayed at 29 GiB.

Proxmox UI → VM 123 → Hardware → Hard Disk (scsi0) → Resize disk.
Set total size to 100 GiB (not 100 TiB, not 34000).
Apply — online grow works for virtio-scsi on Ubuntu 24.04 in most cases.

If / is full, free space before growpart (journal vacuum). Then grow the filesystem with an explicit size so a mis-sized virtio disk cannot expand ext4 to terabytes:

ssh infra-services-cursor 'sudo journalctl --vacuum-size=200M'   # if / is full
ssh infra-services-cursor 'lsblk /dev/sda'
ssh infra-services-cursor 'sudo growpart /dev/sda 1'             # if partition < disk
ssh infra-services-cursor 'sudo resize2fs /dev/sda1 96G'         # explicit target, not bare resize2fs
ssh infra-services-cursor 'df -h /'

Do not run bare resize2fs /dev/sda1 when lsblk shows a multi-terabyte partition — always pass an explicit target such as 96G (or your target minus a few GiB for boot slices).

Adjust partition/device if lsblk shows something other than sda1 (unlikely on this VM).

3. Increase RAM (Proxmox)¶

VM 123 → Hardware → Memory → set 16384 MiB (16 GiB).
Apply — takes effect on next boot or immediately if hotplug is enabled (Proxmox usually requires a reboot for RAM on QEMU VMs).

ssh infra-services-cursor 'sudo reboot'
# wait, then:
ssh infra-services-cursor 'free -h'

4. Enable swap (Ansible)¶

Repo prep (already in tree when merged):

common_swapfile_enabled: true in infra/ansible/inventory/host_vars/infra-services.yml
common role task swapfile.yml

After git pull on the host, ansible-pull applies swap on next converge, or run manually:

ssh infra-services-cursor 'sudo systemctl start ansible-pull-apply.service'
ssh infra-services-cursor 'swapon --show; free -h'

One-time manual fallback (if Ansible has not run yet):

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

5. Reload Prometheus rules¶

After the alert file is on the host (git pull under /var/lib/ansible-pull/homelab):

ssh infra-services-cursor 'docker exec prometheus wget -qO- --post-data="" http://localhost:9090/-/reload'

Or restart the prometheus container from /opt/homelab/services/monitoring.

6. Post-resize verification¶

ssh infra-services-cursor 'free -h && df -h / && docker system df'
ssh infra-services-cursor 'docker stats --no-stream --format "{{.Name}} {{.MemUsage}}" | sort'

Pass criteria:

≥10 GiB free on / at idle
≥4 GiB MemAvailable under normal load (on 16 GiB VM)
Swap active (swapon --show)
Prometheus: no firing InfraServices* alerts after 15 minutes

7. Update inventory and docs¶

Edit inventory/hosts/infra-services.yaml (ram_gb: 16, storage.size_gb: 100).
Run generators and commit.
Optional: python scripts/proxmox-scan.py to refresh compute-live JSON.

Wazuh SIEM — retired; not running on this host
Adding a service — disk budget before new stacks
Coordinated OS patching — patch window load
Synology capacity ntfy — separate NAS alerts

Research updates (2026-06-17)¶

Historical checks from infra-services-cursor plus the frozen Proxmox storage audit prox-storage-2026-06-24.md. These are mixed June capture dates, not a current sizing or scheduling basis. The research instructions and recommendations below are superseded; do not replay them against the later allocation.

Recorded guest state (2026-06-17)¶

Signal	Value	Implication
`/` usage	29 GiB / 29 GiB (~100%, ~33 MiB free)	Blocker for `docker pull`, swapfile, apt
Block device	`sda` 30 GiB → `sda1` 29 GiB ext4 on `/`	Standard Ubuntu 24.04 cloud image layout
Boot partitions	`sda15` EFI 106M, `sda16` `/boot` 913M	Leave untouched during grow
RAM	7.7 GiB total, ~4.1 GiB MemAvailable, no swap	Memory OK today; OOM risk on spikes
Containers	17 running (was 13 pre-Wazuh in guest scan)	+Wazuh×3, +authentik-outpost
`cloud-guest-utils`	installed (`growpart` available)	Guest grow after Proxmox disk resize is one command
QEMU guest agent	running	Clean shutdown from Proxmox UI works
`docker image prune -f`	0 B reclaimed	All 17 images referenced by running containers

Docker disk (docker system df):

Layer	Size	Notes
Images	13.32 GiB (17 active)	Wazuh trio ~7 GiB (indexer 2.49 + manager 2.44 + dashboard 2.08 GiB)
Container writable layers	3.00 GiB	`wazuh-manager` alone ~3 GiB — unusually large; watch on upgrades
Named volumes	3.07 GiB	Prometheus 1.5 GiB, AdGuard 559 MiB, Mongo 507 MiB, Loki 318 MiB, Wazuh queue 270 MiB, indexer data 3 MiB
Build cache	0	—

Top RSS (steady state today):

Container	RSS
wazuh-indexer	1.39 GiB
wazuh-manager	515 MiB
ara	287 MiB
wazuh-dashboard	199 MiB

Non-Docker disk hog: systemd journal /var/log/journal ~2.8–2.9 GiB — largest single reclaimable chunk without stopping stacks.

Proxmox side (VM 123, audit 2026-06-24)¶

Item	Value
VMID / name	123 / `infra-services`
Datastore	`local-lvm` (thin LVM on prox NVMe)
Provisioned disk	30 GiB (`vm-123-disk-1`; ~65% LVM data at audit → guest now ~100%)
Provisioned RAM	8192 MiB
Thin pool headroom	~271 GiB free on 794 GiB pool (67.5% used at audit)
Hypervisor RAM	prox 60 GiB total — +8 GiB for VM 123 is trivial

Verdict: Proxmox storage is not the blocker; the guest filesystem is full. The final close-out landed at 100 GiB after the 34 TiB typo disk was corrected.

Patch-controller SSH from infra-services → prox returned permission denied during this research pass — perform Proxmox changes from the Proxmox UI or a workstation with prox admin access, not via the patch-orchestrator key.

Proxmox resize — exact targets¶

Disk (UI): VM 123 → Hardware → Hard Disk (scsi0) on local-lvm → Resize → 100 GiB total.

Disk (CLI on prox):

qm resize 123 scsi0 +70G    # 30 → 100 GiB
# verify
qm config 123 | grep scsi

Guest grow (after Proxmox resize, online OK for virtio-scsi):

ssh infra-services-cursor 'lsblk /dev/sda'
ssh infra-services-cursor 'sudo growpart /dev/sda 1 && sudo resize2fs /dev/sda1 96G && df -h /'

Expected: sda shows 100G, sda1 grows to ~96G, / reports ~60+ GiB free at current utilization.

RAM (UI): VM 123 → Hardware → Memory → 16384 MiB.

RAM (CLI on prox):

qm set 123 --memory 16384
qm reboot 123    # RAM change requires reboot on this VM type

Immediate relief before Proxmox (no stack stop)¶

If you need headroom today without a hypervisor change:

# ~2.5 GiB back from journal (safe on homelab; logs also in Loki)
ssh infra-services-cursor 'sudo journalctl --vacuum-size=200M && df -h /'

That may free enough space for a 4 GiB swapfile until the proper disk resize. Do not rely on journal vacuum long term — fix root size.

Still cannot docker image prune meaningfully while all 17 images are in use.

Monitoring¶

Alert	Threshold	Firing today?
`InfraServicesDiskCritical`	<10% free on `/`	Yes (deployed via hotfix + in repo)
`InfraServicesDiskPressure`	<15% free	Yes
`InfraServicesMemoryLow`	MemAvailable <1.5 GiB	No (~4.1 GiB available)
Generic `DiskSpaceLow` (all hosts)	<20% free	Also fires for infra-services

After resize + git pull, ansible-pull enables swap; reload Prometheus if alert file was only hotfixed:

ssh infra-services-cursor 'docker exec prometheus wget -qO- --post-data="" http://localhost:9090/-/reload'

Historical maintenance order (superseded)¶

Optional: journal vacuum if you need swap before Proxmox.
Proxmox: disk 100 GiB → guest growpart + explicit resize2fs.
Proxmox: RAM 16 GiB → reboot VM.
git pull on host → ansible-pull (swapfile) → verify swapon --show.
Update inventory/hosts/infra-services.yaml + generators + commit.
Post-check: df -h /, free -h, confirm InfraServices* alerts clear.

Open follow-ups (not blocking resize)¶

Wazuh — stack, agents, and Docker volumes removed. Historical restic snapshots were left in place.
Refresh prox live scan after resize: python scripts/proxmox-scan.py (updates compute-live/ JSON).

Troubleshooting: resize failures (2026-06-17 incident)¶

Symptoms¶

mkdir: cannot create directory '/tmp/growpart.…': No space left on device
resize2fs: Block bitmap checksum does not match bitmap
lsblk shows sda / sda1 as 34T
df shows / still 29G

Root causes¶

/ at 100% — growpart needs /tmp; vacuum journal first.
Proxmox disk set to ~34 TiB instead of 100 GiB — partition auto-span matched the bogus virtio size; bare resize2fs then tried to grow toward TiB and failed on a full filesystem.
resize2fs abort — read-only e2fsck -n reported clean after journal vacuum; no offline fsck required in this case.

Recovery (reported June outcome; non-replayable history)¶

Non-replayable history — do not run

These lines were recorded as successful at the time; they do not validate the contradictory partition/LV sequence below. A read-only filesystem check of a mounted root is not an offline integrity or recovery acceptance test.

# DO NOT RUN: sudo journalctl --vacuum-size=200M          # freed ~2.5 GiB
# DO NOT RUN: sudo e2fsck -n /dev/sda1                    # verify clean before retry
# DO NOT RUN: sudo resize2fs /dev/sda1 96G              # explicit size — NOT bare resize2fs
# DO NOT RUN: df -h /                                     # expect ~96G size after final close-out

Quarantined shrink transcript (June 2026; non-replayable history)¶

Unsafe historical sequence — do not run

The written 96G filesystem followed by an 85GiB partition endpoint can truncate filesystem data. The forced 100G LV reduction also conflicts with the later recorded 200 GiB disk. The record does not establish that this exact contradictory sequence ran, nor that present corruption exists. Neither the reported close-out nor a snapshot makes this a safe procedure.

Proxmox reported that disk shrinking was unsupported. The original notes below, including their “Completed fix” label and cloud-init setting, are preserved solely as a transcript. Every line is a shell comment to prevent direct replay. There is no outstanding shrink task and no replacement shrink recipe here. Do not infer current cloud-init settings from this transcript. Use only the grow-only guidance for future approved expansion.

# DO NOT RUN: **Completed fix (2026-06-25, VM 123 → 100 GiB):**
# DO NOT RUN:
# DO NOT RUN: 1. Snapshot: `qm snapshot 123 pre-disk-shrink --description "…"`
# DO NOT RUN: 2. Guest (online): disable autogrow → shrink partition → shutdown
# DO NOT RUN: 3. Proxmox (VM stopped): `lvresize -L 100G -f /dev/pve/vm-123-disk-1`
# DO NOT RUN: 4. Fix GPT backup header: `sgdisk -e /dev/pve/vm-123-disk-1` (required after
# DO NOT RUN:    `lvresize` or the VM will not get network)
# DO NOT RUN: 5. Reattach disk if needed: `qm set 123 --scsi0 local-lvm:vm-123-disk-1,…,size=100G`
# DO NOT RUN: 6. Boot guest: `growpart /dev/sda 1 && resize2fs /dev/sda1`
# DO NOT RUN: 7. Delete snapshot: `qm delsnapshot 123 pre-disk-shrink`
# DO NOT RUN:
# DO NOT RUN: **Disable cloud-init autogrow** (Ansible: `common_cloud_init_disable_autogrow: true`):
# DO NOT RUN:
# DO NOT RUN: [original configuration block]
# DO NOT RUN: # /etc/cloud/cloud.cfg.d/99-disable-autogrow.cfg
# DO NOT RUN: growpart:
# DO NOT RUN:   mode: off
# DO NOT RUN: resize_rootfs: false
# DO NOT RUN: [end original block]
# DO NOT RUN:
# DO NOT RUN: **Guest shrink (before hypervisor):**
# DO NOT RUN:
# DO NOT RUN: [original shell block]
# DO NOT RUN: yes Yes | sudo parted ---pretend-input-tty /dev/sda unit GiB resizepart 1 85GiB
# DO NOT RUN: sudo partprobe /dev/sda
# DO NOT RUN: lsblk /dev/sda    # sda1 ~84G while sda may still show 34T until step 3–4
# DO NOT RUN: sudo shutdown -h now
# DO NOT RUN: [end original block]
# DO NOT RUN:
# DO NOT RUN: **Proxmox shrink (from prox shell, VM stopped):**
# DO NOT RUN:
# DO NOT RUN: [original shell block]
# DO NOT RUN: lvresize -L 100G -f /dev/pve/vm-123-disk-1
# DO NOT RUN: sgdisk -e /dev/pve/vm-123-disk-1
# DO NOT RUN: qm set 123 --scsi0 local-lvm:vm-123-disk-1,discard=on,ssd=1,size=100G
# DO NOT RUN: qm start 123
# DO NOT RUN: [end original block]
# DO NOT RUN:
# DO NOT RUN: Until steps 3–4 complete, `lsblk` may still show a bogus **34T** `sda` while
# DO NOT RUN: `sda1` is already sane — fix the hypervisor LV, not just the partition.

Close-out (2026-06-25)¶

The June close-out reported the 34 TiB typo disk fully corrected on VM 123. These outcomes are preserved as recorded, not validation of the unsafe transcript:

Check	Result
Proxmox `scsi0`	100 GiB (`vm-123-disk-1`)
Guest `lsblk`	`sda` 100G, `sda1` 99G
ext4 `/`	96 GiB, ~62 GiB free
RAM	16 GiB
Swap	4 GiB `/swapfile`
Stacks	17/17 containers healthy
Wazuh agents	prox, infra-services, saltierpoop Active
Snapshot `pre-disk-shrink`	removed (reclaimed thin-pool headroom)
cloud-init autogrow	disabled via Ansible

That close-out was superseded by the July resize.

Close-out (2026-07-29)¶

Check	Result
Proxmox CPU	12 vCPU active
Proxmox memory	28 GiB active
Proxmox `scsi0`	200 GiB
Guest ext4 `/`	190 GiB, ~115 GiB free
Guest memory	~18 GiB available after startup
Swap	4 GiB, unused
Existing containers	All health checks green
Wazuh	Indexer, manager, and dashboard healthy after TLS ownership repair
Proxmox `local-lvm`	~321 GiB available

Inventory now records 12 vCPU, 28 GiB RAM, and 200 GiB storage.