Central syslog — Graylog (LXC 109)

Role: Central syslog for network gear and unmanaged hosts — not SIEM. Host: prox LXC 109 (graylog, 192.168.6.197, graylog.lab.local) Web UI: https://graylog.infra.realemail.app (infra Traefik + Authentik) SIEM: Wazuh on prox, infra-services, saltierpoop. Container ops logs: Loki + Alloy on infra-services (Promtail retired 2026-07-27).

---

Non-goals

• Do not configure Graylog as primary security correlation (use Wazuh).
• Do not forward full managed-Linux syslog to Graylog and run Wazuh agents collecting the same auth logs (Pattern E).

---

Current state (2026-06-26)

Check	Status
LXC 109 power	running on prox
IP	192.168.6.197/24 (Servers VLAN)
Disk	50 GB rootfs (grown 2026-08-03; was 30 GB — journal preflight OOM)
Journal	message_journal_max_size = 2gb
Graylog	6.1.11, graylog-server + mongod active
Syslog input	UDP 514 (Graylog/java); TCP 514 not bound on LXC
Web UI	HTTPS https://graylog.infra.realemail.app (Traefik + Authentik); backend 0.0.0.0:9000
Remote senders	Complete — UDM, UniFi, prox, harbor, pulse, octoprint, Whrrr, HAOS
Rollout	Done (Pattern E central syslog)

Start/stop from prox:

pct start 109
pct stop 109
pct status 109

---

Log routing model (Pattern E)

Layer	Tool	Sources
SIEM / auth / FIM	Wazuh	prox, infra-services, saltierpoop
Container + host ops	Loki	infra-services Docker stacks
Network + appliances + unmanaged	Graylog	UDM, UniFi, NAS, LXCs without Wazuh, etc.

---

Syslog sender matrix

Target for all Graylog senders: 192.168.6.197:514 UDP (TCP 514 is not listening on LXC 109).

Must send (high value)

Source	IP / reachability	How to configure	Owner
UDM SE (gateway + controller)	192.168.1.1 → Servers	See UniFi SIEM (Network 9.x) below	UI
UniFi switches (USW Pro Max 16, Flex Mini)	via UDM	Controller forwards device syslog when UDM SIEM is enabled	UI
UniFi APs (dining, hallway)	via UDM	Same	UI
Proxmox (prox)	192.168.6.71	Datacenter → prox → Syslog → 192.168.6.197:514 — hypervisor events only; do not duplicate guest auth (Wazuh covers prox OS)	UI or /etc/rsyslog.d/

UDM/UniFi is the bulk of the value — firewall events, DHCP, DNS filter, WiFi, adoption, IDS/IPS if enabled.

UniFi SIEM (Network 9.x)

UniFi splits syslog across two UI locations; configure both to 192.168.6.197:514 (UDP):

Location	Typical contents
Settings → Control Plane → Integrations → SIEM Server	Admin, config, platform events
Settings → CyberSecure → Traffic Logging → SIEM Server	Firewall / flow / security

Under Contents → Edit Selections, enable at least Firewall Default Policy and Security Detections (empty selection = no forward).

Zone-based firewall: each ZBF rule that should appear in Graylog needs Log enabled on the rule itself — SIEM on alone does not emit per-rule hits.

Graylog source is often the device hostname (e.g. UDM-SE), not udm or 192.168.1.1.

Should send (unmanaged Servers VLAN)

Source	VMID / IP	Notes
harbor-registry	119 / Servers	Running; no Wazuh agent
pulse	116 / 192.168.6.199	Pulse Monitoring Server (PVE dashboard)
octoprint	120 / 192.168.6.222	3D printer control (privileged USB)
Whrrr (Synology DSM)	192.168.6.215	Log Center → syslog 192.168.6.197:514 UDP
HAOS (poopcastle)	200 / 192.168.6.227	mib1185/ha-addon-syslog add-on → UDP 514 (manual)

Configure with rsyslog (*.* @192.168.6.197:514) or app-native syslog settings.

Ansible (LXCs on prox): role graylog-syslog-client — converges automatically on prox ansible-pull (site.yml → hypervisors play). Manual playbook only for bootstrap / -e graylog_syslog_send_test_message=true:

# on infra-services (sudo — patch-controller key is root-only)
sudo ansible-playbook playbooks/graylog-syslog-clients.yml \
  -i inventory/generated.yml -l prox

Targets: harbor-registry (119), pulse (116), octoprint (120; starts LXC if stopped).

HAOS (VM 200) — manual add-on

HAOS has no writable rsyslog. Use the mib1185/ha-addon-syslog add-on (owner UI — not Ansible). Done 2026-06-26.

1. Settings → Add-ons → Add-on store → ⋮ → Repositories → add https://github.com/mib1185/ha-addon-syslog
2. Install Syslog, configure:

syslog_host: 192.168.6.197
syslog_port: 514
syslog_protocol: udp
syslog_ssl: false
syslog_ssl_verfify: false

1. Start add-on; enable Start on boot.
2. Verify in Graylog: search poopcastle, haos, or source 192.168.6.227.

No firewall change — HAOS and Graylog are both on Servers VLAN.

Do not send (already covered elsewhere)

Source	Reason
infra-services	Wazuh agent + Loki/Alloy
saltierpoop	Wazuh agent
prox OS auth/syscheck	Wazuh agent — Proxmox platform syslog OK
Security cameras (G4/G5)	UniFi Protect on UDM; no direct syslog
IoT / Appliances VLAN	Isolated; most devices cannot/should not syslog to Servers

Low priority / stopped experiments

mysql, ollama, penpot, caddy, metrimon, etc. — configure only if revived. ~~sqlserver2022 (110)~~ — retired 2026-06-25.

---

Firewall (ZBF)

Graylog listens on Servers VLAN (192.168.6.197).

Flow	Needed?	Notes
Gateway (UDM) → Servers :514	Yes	Zone matrix: Gateway → Internal ALLOW
GenPop/Personal → Servers :514	Only if a host sends directly	GenPop/Personal → Servers already ALLOW (all ports)
IoT/Security → Servers :514	No (default deny)	By design

No new ZBF rule required for UDM remote logging in the current live posture (firewall-live.md).

---

Rollout phases

Phase A — Graylog healthy (done / verify)

1. ~~pct start 109~~
2. Confirm listeners: pct exec 109 -- ss -ulnp | grep 514
3. Open UI: https://graylog.infra.realemail.app (Authentik) — or LAN debug http://192.168.6.197:9000. Admin password on infra-services: /etc/homelab/graylog-admin.password (1Password copy recommended)
4. Confirm existing Syslog UDP/TCP input in Graylog (System → Inputs)

Phase B — Network gear (done)

1. ~~UDM SIEM Server → 192.168.6.197:514 UDP~~ (2026-06-25)
2. ~~Enable log categories + ZBF per-rule logging~~
3. ~~Proxmox datacenter syslog~~ (syslog: 192.168.6.197 on prox)
4. Verify in Graylog: search by hostname (UDM-SE, prox.usg.realemail.app) or browse Sources

Test from infra-services (UDP — matches Graylog listener):

logger -n 192.168.6.197 -P 514 -d "graylog-udp-test $(hostname)"

Phase C — Unmanaged Servers hosts

Automatic: prox ansible-pull applies graylog-syslog-client on each converge (harbor-registry, pulse, octoprint when running).

Manual bootstrap (optional):

sudo ansible-playbook playbooks/graylog-syslog-clients.yml \
  -i inventory/generated.yml -l prox

Manual fallback (Debian/Ubuntu LXC):

echo '*.* @192.168.6.197:514' | sudo tee /etc/rsyslog.d/90-graylog.conf
sudo systemctl restart rsyslog

Whrrr: Control Panel → Log Center → Log Sending → 192.168.6.197:514 UDP.

HAOS: mib1185/ha-addon-syslog add-on (see HAOS section).

Phase D — Close-out (done)

1. ~~Update sender verification log~~ (2026-06-26 — all senders verified)
2. Optional: Graylog streams/extractors — see Optional hygiene below
3. ~~proxmox.power: running in inventory~~
4. ~~README Owner TODO → :white_check_mark:~~ (2026-06-25)
5. ~~python scripts/proxmox-scan.py refresh compute-live~~ (2026-06-25)

---

Sender verification log

Source	Configured	Verified in Graylog	Date
UDM SE	✅ SIEM + ZBF logging	✅	2026-06-25
UniFi switches/APs	✅ via UDM + direct	✅ 192.168.6.1 (UDM), 192.168.1.58 (USW Pro Max)	2026-06-25
Proxmox (prox)	✅ rsyslog + cluster syslog	✅ prox.usg.realemail.app	2026-06-25
harbor-registry	✅ Ansible graylog-syslog-client	✅	2026-06-25
pulse	✅ Ansible graylog-syslog-client	✅	2026-06-25
octoprint	✅ Ansible (auto-start LXC)	✅ 192.168.6.222 → UDP 514	2026-06-25
Whrrr (DSM)	✅ Log Center UDP	✅ whrrr	2026-06-25
HAOS (poopcastle)	✅ mib1185 syslog add-on	✅ owner verified	2026-06-26

---

Optional hygiene

Pattern E rollout is complete — all senders verified. Streams configured 2026-06-26 (search hygiene by source regex; no extractors required for basic filtering).

Stream	source regex (summary)
udm	UDM-SE, 192.168.1.1
unifi-devices	USW, UAP, UniFi, 192.168.1.58, 192.168.6.1
proxmox	prox.usg.realemail.app, proxbox, 192.168.6.71
synology	whrrr, 192.168.6.215

Legacy streams from the pre-homelab install (UniFi, Pihole, Unifi IPFIX) remain in Graylog but are unused by Pattern E — safe to disable in the UI when cleaning up.

Admin UI: https://graylog.infra.realemail.app (Traefik + Authentik) — password on infra-services at /etc/homelab/graylog-admin.password (store a copy in 1Password). Direct http://192.168.6.197:9000 is LAN/debug only.

Re-apply or extend streams:

# on infra-services (uses patch-controller SSH to prox LXC 109)
GRAYLOG_ADMIN_PASSWORD=$(sudo cat /etc/homelab/graylog-admin.password) \
  bash /opt/homelab/scripts/configure-graylog-streams.sh

Script in repo: scripts/configure-graylog-streams.sh.

---

Troubleshooting

Symptom	Check
Nothing in Graylog	Input enabled? ss -ulnp \| grep 514 on LXC 109
UI connection refused on :9000	graylog-server active? Disk free ≥ journal max (df -h; message_journal_max_size)? Crash-loop in journalctl -u graylog-server
HTTPS UI fails	Traefik dynamic graylog.yml present? Authentik outpost healthy? dig graylog.infra.realemail.app → 192.168.6.17
UDM not arriving	Both SIEM locations configured? Edit Selections non-empty? UDP not TCP? tcpdump on LXC: dst host 192.168.6.197 and udp port 514
UDM SIEM on, no firewall events	ZBF rules need Log enabled per rule; enable Firewall Default Policy in Contents
Proxmox not arriving	Datacenter syslog server saved; test with logger from prox
Duplicate noise	Confirm Wazuh hosts are not in rsyslog forward list
HAOS not arriving	Add-on running? UDP not TCP; tcpdump on LXC: host 192.168.6.227 and udp port 514

---

Related

• Compute disposition review — Pattern E
• Wazuh SIEM
• Device VLAN mapping
• Firewall (live)
• Home Assistant OS (haos)