Skip to content

Check-in — 2026-07-27

GitHub Actions runner observability went from design to a live fleet, and homelab CI capacity stopped serializing on a single runner slot.

TL;DR

Central monitoring on infra-services now accepts authenticated Alloy remote write/logs from every Linux runner placement group. Six repo-scoped homelab-ci-* runners absorb Lint fan-out. Komodo DeployStack no longer fails on the local collector image, and Hyper-V NAT DNS/ACL gaps that blocked fleet telemetry are fixed in Ansible. The central runner-management suite now adds a manifest-driven, dry-run-first path for fleet observation and guarded repair.

Phase status

Area Status Notes
GHA observability stack Live Alloy receive + collector + dashboards/alerts (#135–#138)
Monitoring SOPS / telemetry auth Live GitHub App + per-host Traefik basic-auth clients (#137)
Runner fleet canary → full roll Live general-a/b, docker-a/b, lan-a all up in Prometheus (#142)
Homelab CI capacity Live homelab-ci-01–06 (3 on A, 3 on B) (#140)
Komodo monitoring DeployStack Unblocked pull_policy: never for local collector (#139, #141)
Promtail → Alloy central logs Live Promtail retired; Docker/syslog continuity verified in Alloy (#144)
Organization workflow/runner audit Closed 24 repos / 68 workflows audited; managed runners, watchdogs, and publish-only aiProject verified (audit)
Central runner management Implemented Dispatcher + reusable observe/qualify/watchdog/maintenance workflows, scheduled checks, canonical manifest, and guarded adapters
Service validation TODOs Closed Paperless SOPS/GPT, Stirling merge/split/OCR, and LiteLLM→Phoenix trace live-verified

What shipped

  • Runner observability role + installers, Grafana dashboards A–E, Prometheus alerts, collector binary, telemetry Traefik routes (github-actions-observability.md)
  • Canary on gha-general-a, then fleet roll with /etc/hosts telemetry pin and named/default _diag ACLs for Hyper-V NAT VMs
  • Six homelab-ci-* instances; fleet/inventory/runner-health expected list expanded to seventeen Linux runners with aiProject general/Docker classes
  • Komodo-safe local image handling (pull_policy: never) after build forced Go module fetches over broken host IPv6 DNS
  • Promtail retired after Alloy Docker/syslog cutover; first-start stale history is dropped before Loki's reject-old-samples boundary
  • Organization-wide workflow audit: 22 legacy runners retired, aiProject moved to class-scoped runner selectors, and GitHub-hosted stale-run watchdogs merged and dry-run verified across all six self-hosted repositories
  • aiProject's stale SSH deploy was retired; its publish-only main run passed, obsolete deploy credentials were deleted, and GPG-enabled runner image r4 was rolled to all five hosts
  • Central GitHub runner management shipped with GitHub-hosted API discovery, exact-target qualification, protected watchdog/maintenance applies, ansible@v1, and 30-day portable reports
  • Reconciled stale service TODOs: Paperless has 41 documents and live GPT processing, Stirling passed merge/split/OCR operations, and LiteLLM exported a three-span, 64-token LLM trace into Phoenix
  • Fixed infra-services Docker bridge DNS by using local AdGuard with public fallbacks; the exact fresh Go module repro dropped from a 21-second timeout to a one-second success
  • Installed an Ansible-owned Paperless correspondent prompt with a strict entity-only / 128-character output contract; monitoring GitHub App key rotated

Milestone delta: central runner management

The suite consolidates fleet operations behind one dispatcher while keeping API control on GitHub-hosted workers and host mutation on a separate Ansible-capable management runner. Applied work remains owner-authorized, scope-bounded, state-rechecked, and protected by environment approval.

flowchart LR
  D["Central dispatcher"] --> R["Reusable operation"]
  R --> P["Dry-run plan / report"]
  P --> A["Owner + protected approval"]
  A --> M["Exact target via ansible@v1"]
  M --> V["Observe + qualify"]

Known gaps / drift

  • Hyper-V VM dynamic memory may still under-assign under heavy parallel CI; watch Assigned vs Max during Lint storms

What remains

Item Owner Priority
Configure runner-fleet Apps, environments, and management-runner variable Owner P1
Monitor Hyper-V Assigned RAM during the next CI storm Agent + Owner P2

Mermaid: runner telemetry path

flowchart LR
  subgraph hyperv ["Hyper-V NAT runners"]
    A["gha-general-a/b"]
    D["gha-docker-a/b"]
  end
  subgraph lan ["LAN"]
    L["infra-services lan-a Alloy"]
  end
  A -->|hosts pin + HTTPS basic-auth| T["telemetry.infra.realemail.app"]
  D --> T
  L --> T
  T --> Alloy["central Alloy receive_http / Loki API"]
  Alloy --> Prom["Prometheus"]
  Alloy --> Loki["Loki"]
  Coll["github-actions-collector"] --> Prom