Skip to content

Check-in — 2026-07-27

GitHub Actions runner observability went from design to a live fleet, and homelab CI capacity stopped serializing on a single runner slot.

TL;DR

Central monitoring on infra-services now accepts authenticated Alloy remote write/logs from every Linux runner placement group. Six repo-scoped homelab-ci-* runners absorb Lint fan-out. Komodo DeployStack no longer fails on the local collector image, and Hyper-V NAT DNS/ACL gaps that blocked fleet telemetry are fixed in Ansible.

Phase status

Area Status Notes
GHA observability stack Live Alloy receive + collector + dashboards/alerts (#135–#138)
Monitoring SOPS / telemetry auth Live GitHub App + per-host Traefik basic-auth clients (#137)
Runner fleet canary → full roll Live general-a/b, docker-a/b, lan-a all up in Prometheus (#142)
Homelab CI capacity Live homelab-ci-01–06 (3 on A, 3 on B) (#140)
Komodo monitoring DeployStack Unblocked pull_policy: never for local collector (#139, #141)
Promtail → Alloy central logs Open Promtail still unhealthy; Alloy log cutover deferred
Owner product TODOs Open Paperless SOPS/GPT, Stirling smoke, LiteLLM→Phoenix trace proof

What shipped

  • Runner observability role + installers, Grafana dashboards A–E, Prometheus alerts, collector binary, telemetry Traefik routes (github-actions-observability.md)
  • Canary on gha-general-a, then fleet roll with /etc/hosts telemetry pin and named/default _diag ACLs for Hyper-V NAT VMs
  • Six homelab-ci-* instances; fleet/inventory/runner-health expected list updated to fifteen Linux runners
  • Komodo-safe local image handling (pull_policy: never) after build forced Go module fetches over broken host IPv6 DNS

Known gaps / drift

  • Promtail remains in the monitoring stack and reports unhealthy; Alloy central Docker/syslog cutover still pending (runbook)
  • Host IPv6 DNS to proxy.golang.org is still broken on infra-services — any future collector rebuild needs an explicit offline/vendored path or DNS fix
  • Hyper-V VM dynamic memory may still under-assign under heavy parallel CI; watch Assigned vs Max during Lint storms
  • GitHub App PEM / infra age key were briefly exposed in agent terminals earlier in this work stream — rotate when convenient

What remains

Item Owner Priority
Paperless .env.sops.yaml + GPT profile smoke Owner P0
Stirling-PDF Authentik smoke Owner P1
Confirm LiteLLM OTLP spans in Phoenix Owner P1
Alloy central log cutover; retire Promtail Agent P1
Fix infra-services IPv6/DNS for Go module builds Agent P2
Rotate monitoring GitHub App key if still the exposed one Owner P1

Mermaid: runner telemetry path

flowchart LR
  subgraph hyperv ["Hyper-V NAT runners"]
    A["gha-general-a/b"]
    D["gha-docker-a/b"]
  end
  subgraph lan ["LAN"]
    L["infra-services lan-a Alloy"]
  end
  A -->|hosts pin + HTTPS basic-auth| T["telemetry.infra.realemail.app"]
  D --> T
  L --> T
  T --> Alloy["central Alloy receive_http / Loki API"]
  Alloy --> Prom["Prometheus"]
  Alloy --> Loki["Loki"]
  Coll["github-actions-collector"] --> Prom