Skip to content

Wazuh reboot-persistence fix and deploy drift (2026-07-08)

TL;DR

The recurring "Wazuh dashboard is broken again after a reboot" outage finally has a root cause and a permanent fix (PR #83): it was never TLS — the bind-mounted dashboard runtime config kept drifting between unreadable (EACCES) and writable (duplicate hosts:), and a running container masked the problem until the next restart re-read from disk. The fix makes the file read-only to the container (1000:1000 0440 + :ro mount) and self-heals ownership via ansible-pull. Merging it also exposed a second issue: live edits left on /opt/homelab tracked files blocked the Komodo deploy fast-forward, which is now cleaned.

Phase status

Area Status Evidence
Wazuh SIEM stability Root-caused and fixed PR #83; live restart verified healthy, single hosts: block, no EACCES/duplicate-key
Wazuh runtime-config ownership Deterministic + self-healing sync-wazuh-yml.sh forces 1000:1000 0440; ansible-pull re-asserts on boot + ~30 min
Komodo deploy reliability Unblocked, drift lesson noted komodo-safe-pull.sh refused FF due to host-local edits; tracked files reset, deploy re-run
Backups (PR #80/#81) Merged to main Restic alert scoping + AdGuard upstream stability landed ahead of this work

What shipped

  • Wazuh dashboard permanent fix (PR #83):
  • services/wazuh/scripts/sync-wazuh-yml.sh renders config/wazuh_dashboard/wazuh.runtime.yml as 1000:1000 mode 0440 via inode-safe, in-place writes (root / passwordless-sudo / write-through-container ladder) — no writer can leave it owned by host someone (uid 1001) or writable by the container.
  • services/wazuh/compose.yml mounts the runtime config :ro as defense in depth against the dashboard appending its own default hosts: entry.
  • roles/common/tasks/wazuh-secrets.yml now re-asserts cert ownership (fix-cert-permissions.sh) and re-renders the runtime config on every ansible-pull (boot + ~30 min timer, infra-services only), so drift self-heals without a manual step.
  • README and wazuh-siem.md document both failure modes and the read-but-not-write model.
  • Deploy unblock: reset four host-local tracked-file modifications (backups/restore-test.sh, infra/ansible/inventory/files/backup-fetch.pub, infra/ansible/inventory/group_vars/all/backup.sops.yaml, monitoring/prometheus/alerts/backup.yml) whose contents already matched origin/main, allowing /opt/homelab to fast-forward to 17a7800 and the Komodo webhook to fire.

Known gaps / drift

  • Live edits on the deploy target are a process smell. Backup-hardening work left tracked files modified in /opt/homelab; because they matched what was later merged, no content was lost, but komodo-safe-pull.sh only archives untracked blockers — it aborts on modified tracked files. Future live patching should go through a branch + merge, not direct edits to the deploy checkout.
  • Reboot proof is still pending a real reboot. The fix was verified with a container restart (the actual trigger), which is equivalent, but the next planned host reboot is the true confirmation that ansible-pull self-heal + :ro mount hold.
  • The dashboard healthcheck only probes its own process, so it can report healthy while the indexer-backed config is broken — a stricter check was considered but deferred to avoid restart flapping.

What remains

Item Owner Priority
Confirm Wazuh dashboard stays healthy across the next real host reboot Owner + Agent P0
Verify the ansible-pull Wazuh re-render task runs green on infra-services Agent P1
Stop live-editing tracked files on /opt/homelab; branch + merge instead Owner + Agent P1
Consider a dashboard healthcheck that reflects backend config load Agent P2

Mermaid: the reboot failure loop and the fix

flowchart TD
    R["Restart / reboot re-reads wazuh.runtime.yml from disk"] --> Q{"On-disk state?"}
    Q -->|"1001:1001 / 0600"| E["Dashboard cannot read -> EACCES"]
    Q -->|"writable by uid 1000"| D["Dashboard appends hosts: -> duplicated mapping key"]
    E --> B["Config load fails (looks like cert error)"]
    D --> B
    B --> F["Prior fixes flipped to the other failure mode"]

    F --> FIX["PR #83 permanent fix"]
    FIX --> M1["1000:1000 mode 0440 (read, not write)"]
    FIX --> M2[":ro compose mount"]
    FIX --> M3["ansible-pull re-asserts on boot + 30 min"]
    M1 --> OK["Healthy across restart"]
    M2 --> OK
    M3 --> OK