Skip to content

Prox memory soak (24h attribution)

Temporary sampler on infra-services while Prox physical RAM is already at max DIMM capacity. Goal: 24h of attributed samples, then shrink or retire guests (not buy RAM).

Policy (owner Q&A 2026-08-09)

Decision Choice
Extra DIMMs Impossible — chassis at max
Attribution Deep: host + guests + top ps inside any guest with RSS-ish mem > 1 GiB
Discord during soak NodeMemoryLow already notify: ticket (no Discord page)
Cadence Hourly always + extra sample when NodeMemoryLow{instance=prox} fires (15m cooldown)
Artifact home /var/log/homelab/prox-mem-soak/ on infra-services
End decision Shrink/retire guests

Units

  • homelab-prox-mem-soak.timer — hourly
  • homelab-prox-mem-soak-alert.timer — poll every 5m with --if-alert
  • homelab-prox-mem-soak-finalize.timer — one-shot at end of 24h window (2026-08-10 11:15 America/Los_Angeles): writes REPORT.md + high-priority ntfy to homelab-prox-mem-soak with decision prompt
  • Scripts: scripts/prox-mem-soak-sample.sh, scripts/prox-mem-soak-finalize.sh (ansible-pull path /opt/homelab/scripts/...)
  • Ansible role: prox-mem-soak (infra-services via site.yml)

Phone subscribe (finalize notify)

Subscribe ntfy app to topic homelab-prox-mem-soak on ntfy.sh before Mon 2026-08-10 11:15 PDT. Smoke:

curl -d "prox mem soak finalize channel smoke" \
  -H "Title: Prox soak ntfy test" \
  -H "Priority: high" \
  "https://ntfy.sh/homelab-prox-mem-soak"

Manual finalize (idempotent; use --force to redo):

sudo /opt/homelab/scripts/prox-mem-soak-finalize.sh
sudo less /var/log/homelab/prox-mem-soak/REPORT.md

Manual sample

sudo /opt/homelab/scripts/prox-mem-soak-sample.sh --reason manual
sudo ls -lt /var/log/homelab/prox-mem-soak/$(date -u +%Y-%m-%d)/
sudo jq '{ts, reason, memory, swap, heavy: [.heavy_process_samples[] | {name, mem_bytes, ps_method}]}' \
  /var/log/homelab/prox-mem-soak/latest.json

After 24h

  1. Review samples under /var/log/homelab/prox-mem-soak/.
  2. Pick shrink/retire targets (fat guests today: saltierpoop, infra-services, haos, graylog).
  3. Set prox_mem_soak_enabled: false (or remove role) and stop timers.
  4. Record decision in journal.
flowchart LR
  T1[hourly timer] --> S[prox-mem-soak-sample.sh]
  T2[alert poll 5m] --> S
  S --> Prox[SSH prox pvesh/free/ARC]
  S --> Deep[ps in guests over 1GiB]
  S --> Log["/var/log/homelab/prox-mem-soak"]
  Log --> Decide[shrink or retire guests]