Skip to content

July reliability and observability check-in (2026-07-06)

TL;DR

Since the 2026-07-03 Komodo reliability audit, the repo shifted from incident cleanup into follow-through: Komodo deploy safety, Wazuh readiness, HAOS backup routing, and LLM observability all moved materially forward. The core GitOps spine is still healthy, but this was not a clean PLAN.md phase boundary; it was a dense hardening pass across already-live systems. The main open evidence gaps are Phoenix trace ingestion after the LiteLLM callback fix, Komodo alerting, and external restore proof.

Phase Status

Area Status Evidence
Komodo deploy reliability Improved, still needs alerting Safe pull wrapper, host-local webhook secret, RunSync, and resources.toml CI are merged; Komodo Alerter remains owner action
Wazuh SIEM Operationally ready in README, restore evidence still open Readiness gates marked complete; SEC-016 still tracks restore evidence gaps
LLM observability Live, one trace gate open LiteLLM, Phoenix, Phoenix API key, local Ollama backend, metrics, and dashboard landed; trace export needs post-fix E2E proof
Backups Better HAOS routing HAOS snapshots now prefer NAS with B2 as capped safety net
Docs Drift pass in progress Cloudflare Pages, Wazuh, host lifecycle, service onboarding, and decommission runbooks refreshed in this branch

What Shipped

  • Komodo follow-through: komodo-safe-pull.sh archives untracked host blockers before deploy, the webhook secret is read from host compose.env, and resources.toml validation now catches compose/path drift before merge.
  • Wazuh hardening: post-deploy scripts no longer depend on python3, cert-permission repair runs before security config apply, and generated compose.env ownership was fixed after root SOPS renders.
  • LLM observability: LiteLLM and Phoenix were wired together, LiteLLM secrets gained the Phoenix System API key, the local workstation Ollama backend was smoke-tested through LiteLLM, Prometheus metrics scrape correctly, and the callback registration was corrected for Phoenix tracing.
  • Contractor/security quick wins: Grafana auth-proxy default role dropped to Viewer, dashboard-only commits trigger repo pulls again, ARA/Homepage images are pinned, Promtail positions moved to a named volume, and Tailscale server ACLs narrowed.
  • HAOS backups now route toward the NAS-backed history with B2 kept as a small offsite safety net instead of the primary long-retention path.
  • GitHub Actions moved through the Node 24 transition, including the docs deploy path with wrangler-action@v4.

Known Gaps / Drift

  • SEC-016 remains open for tier-1 external restore evidence and Wazuh restore drill proof. The README readiness row can be true operationally while this evidence gap remains tracked separately.
  • The Wazuh stack is at 4.14.6, but the Ansible wazuh-agent role still defaults agents to 4.14.5; treat the agent bump as a runtime follow-up rather than a docs-only fix.
  • Phoenix trace export is still an explicit owner TODO until a LiteLLM chat completion produces a trace in Phoenix after the callback fix is deployed.
  • Komodo Alerter setup is still owner-only UI work. The repo can reduce failure modes, but it still cannot prove the UI will alert on the next broken deploy.
  • Phoenix is a host-class LXC and not yet represented as a full Komodo-managed server/periphery entry.

What Remains

Item Owner Priority
Run the LiteLLM -> Phoenix full trace test and close the README TODO when traces appear Owner P0
Configure a Komodo Alerter in the UI Owner P0
Capture/record restore evidence for SEC-016 Owner + Agent P1
Decide whether Phoenix should become a full Komodo-managed server in this repo Owner + Agent P1
Continue the docs drift cleanup started in this branch where live validation is required Agent P2

Mermaid: July 3 Audit to Current Gates

flowchart TD
    A["2026-07-03 Komodo audit"] --> B["GitOps reliability fixes"]
    A --> C["Wazuh readiness hardening"]
    A --> D["LLM observability rollout"]
    A --> E["Backup routing improvements"]

    B --> B1["host-local webhook secret"]
    B --> B2["safe pull archives blockers"]
    B --> B3["resources.toml CI validation"]

    C --> C1["post-deploy scripts hardened"]
    C --> C2["readiness row complete"]
    D --> D1["LiteLLM + Phoenix + metrics"]
    D --> D2["callback registration fixed"]
    E --> E1["HAOS NAS-first, B2 capped"]

    B3 --> G["Open gates"]
    C2 --> G
    D2 --> G
    E1 --> G

    G --> H["Komodo Alerter"]
    G --> I["Phoenix trace proof"]
    G --> J["SEC-016 restore evidence"]