Skip to content

GitHub Actions runner observability

This runbook deploys and operates GitHub Actions runner monitoring on the existing infra-services Prometheus, Grafana, Loki, Alertmanager, and Traefik stack. It covers the repository-scoped Linux fleet declared in infra/github-runners/fleet.yml and provides future-ready native Windows onboarding.

Contents

Architecture

flowchart LR
  GH["GitHub App APIs"] --> Collector["GitHub Actions collector"]
  Collector -->|"Prometheus scrape :9464"| Prometheus
  Linux["Linux runner hosts"] -->|"metrics and logs HTTPS push"| Traefik
  Windows["Future Windows runners"] -->|"metrics and logs HTTPS push"| Traefik
  Traefik --> AlloyCentral["Central Alloy"]
  AlloyCentral --> Prometheus
  AlloyCentral --> Loki
  Prometheus --> Grafana
  Loki --> Grafana
  Prometheus --> Alertmanager
  Loki --> Alertmanager

The collector obtains runner state, workflow runs, and workflow jobs from GitHub. Host agents scrape exporters locally, then push to the authenticated Alloy gateway. Neither the Hyper-V runner subnet nor exporter ports need to be routable from infra-services.

Prometheus labels contain bounded dimensions such as repository, normalized workflow/job, runner, placement, status, and conclusion. Run IDs, job IDs, commit SHAs, actors, branches, pull requests, URLs, and errors remain structured Loki fields.

GitHub App

Create a separate read-only App. Do not reuse the runner-registration PAT.

  1. Open GitHub organization settings → Developer settings → GitHub Apps → New GitHub App.
  2. Set the App name to spadoople-runner-observability.
  3. Disable user authorization and webhook delivery; this collector polls.
  4. Set repository permissions:
  5. Actions: Read-only
  6. Administration: Read-only
  7. Metadata: Read-only (automatic)
  8. Set organization permissions:
  9. Self-hosted runners: Read-only
  10. Create the App, generate a private key, and record the App ID.
  11. Install it on spadoople, selecting every repository that should appear in monitoring.
  12. Record the installation ID from the installation URL.

The App installation defines collector scope. No repository list is maintained by hand.

Secrets

Copy the example only when the encrypted file does not already exist:

cd /opt/homelab
test -f services/monitoring/.env.sops.yaml ||
  cp services/monitoring/.env.sops.yaml.example \
    services/monitoring/.env.sops.yaml
sops services/monitoring/.env.sops.yaml

Set the existing monitoring values plus:

GITHUB_APP_ID: "123456"
GITHUB_INSTALLATION_ID: "12345678"
GITHUB_ORG: "spadoople"
# Paste the full App private-key PEM via `sops` (multiline). Do not commit
# plaintext PEMs; the renderer writes a root-only mounted file on the host.
GITHUB_APP_PRIVATE_KEY: "REPLACE_ME_VIA_SOPS"
TELEMETRY_CLIENTS:
  gha_general_a:
    username: "gha-general-a"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  gha_general_b:
    username: "gha-general-b"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  gha_docker_a:
    username: "gha-docker-a"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  gha_docker_b:
    username: "gha-docker-b"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  infra_services:
    username: "infra-services"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"

Commit only SOPS ciphertext. Render on infra-services:

cd /opt/homelab
sudo SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt \
  scripts/render-monitoring-secrets.sh
sudo test -s services/monitoring/.env
sudo test -s services/monitoring/secrets/github-app.pem

The renderer uses restrictive temporary files and atomic installation. It creates the Compose environment, the mounted App key, and telemetry client credential files without printing their contents.

Central deployment

Confirm telemetry.infra.realemail.app resolves to infra Traefik. The endpoint is machine-authenticated and must not use the browser Authentik middleware.

cd /opt/homelab/services/monitoring
docker compose --env-file .env config --quiet
cd /opt/homelab
bash scripts/validate-monitoring.sh

Back up before changing containers:

sudo restic backup \
  /var/lib/docker/volumes/monitoring_grafana-data/_data \
  /var/lib/docker/volumes/monitoring_collector-data/_data

Start the new services while Promtail remains active:

cd /opt/homelab/services/monitoring
docker compose --env-file .env up -d --build github-actions-collector alloy
docker compose ps github-actions-collector alloy prometheus loki grafana alertmanager promtail
docker compose logs --tail=100 github-actions-collector alloy

Recreate backend services only after configuration validation:

docker compose --env-file .env up -d prometheus loki grafana alertmanager

Do not remove Promtail until Alloy has collected the same central Docker and syslog streams without gaps. The committed Compose model retains Promtail for this canary period.

For the cutover, copy the declarations from monitoring/alloy/central-logs.alloy.disabled into config.alloy, validate the result, deploy Alloy, and compare Docker/syslog volume and timestamps for at least one full operating window. Stop Promtail only after continuity is confirmed. Remove the Promtail service and positions volume in a later reviewed change; do not combine that cleanup with the first Alloy deployment.

Linux runner rollout

The authoritative path is Ansible. Schedule this while the selected runner host is idle because adding persistent diagnostic mounts restarts its runner units.

Canary one host:

cd /opt/homelab
ANSIBLE_CONFIG=infra/ansible/ansible.cfg ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml \
  --limit gha-general-a

Verify, then deploy the remaining hosts:

ANSIBLE_CONFIG=infra/ansible/ansible.cfg ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml \
  --limit 'gha-general-b:gha-docker-a:gha-docker-b:infra-services'

For a Linux runner outside Ansible:

sudo bash scripts/runner-observability/install-linux.sh \
  --telemetry-url https://telemetry.infra.realemail.app \
  --username gha-general-a \
  --password-file /secure/path/host-telemetry-password

Only outbound TCP 443 is required. Do not expose node_exporter, cAdvisor, or Alloy listeners to untrusted networks.

Windows runner onboarding

No native Windows runner is currently designated, so this path is statically validated but not considered live-verified.

From an elevated PowerShell 7 session:

Set-Location C:\path\to\homelab
.\scripts\runner-observability\install-windows.ps1 `
  -TelemetryUrl 'https://telemetry.infra.realemail.app' `
  -TelemetryUsername 'windows-runner-host' `
  -TelemetryPasswordFile 'C:\secure\telemetry-password'

The script installs windows_exporter and Alloy as Windows services, validates download hashes, discovers runner services and _diag directories, and creates only outbound telemetry configuration.

Verification

Offline/static checks:

cd /opt/homelab
bash scripts/validate-monitoring.sh

Live checks after the central deployment:

cd /opt/homelab
bash scripts/validate-monitoring.sh --live
docker exec github-actions-collector \
  /github-actions-collector healthcheck http://localhost:9464/readyz
docker exec prometheus wget -qO- \
  'http://github-actions-collector:9464/metrics' |
  grep '^github_actions_runner_online'

After the owner authorizes a notification test, submit the synthetic alert and confirm it reaches the designated test destination:

bash scripts/validate-monitoring.sh --live --synthetic-alert

Run a safe workflow and verify:

  1. The job appears queued on Current GitHub Actions Activity.
  2. runner becomes populated when GitHub assigns it.
  3. Queue and execution durations appear once after completion.
  4. The host dashboard shows the corresponding CPU, memory, disk, network, and runner-container interval.
  5. Runner journal and _diag entries appear in the log dashboard for the same runner and time range.

Dashboards and alerts

Grafana provisions five dashboards in the existing Homelab folder:

  • Runner Fleet Overview
  • Current GitHub Actions Activity
  • Historical Activity and Utilization
  • Runner Host Health
  • Runner Logs and Diagnostics

Prometheus alerts cover runner availability, sustained demand without idle capacity, queue age, queue percentiles, failure rates, inactivity, API health, rate-limit pressure, exporter/collector ingestion, storage, certificate, CPU, memory, disk, latency, and OOM conditions.

Loki rules cover repeated reconnects, runner update failures, and worker exceptions. Existing Discord and ntfy routes remain enabled. Optional Email, Slack, and webhook examples are under monitoring/alertmanager/examples/.

Backup and restore

Grafana and collector state are backed up. Prometheus and Loki remain retention-bounded and regeneratable.

sudo restic backup \
  /var/lib/docker/volumes/monitoring_grafana-data/_data \
  /var/lib/docker/volumes/monitoring_collector-data/_data

Restore into a temporary directory first:

sudo restic restore latest --target /tmp/restore-monitoring
cd /opt/homelab/services/monitoring
docker compose stop grafana github-actions-collector

After verifying the resolved backup paths, restore each volume using the existing monitoring procedure in Restore, then:

docker compose --env-file .env up -d grafana github-actions-collector
docker compose logs --tail=100 grafana github-actions-collector

Loss of collector SQLite does not lose GitHub data, but the next lookback can re-observe completed events and temporarily distort counters. Restore it when available.

Upgrade

  1. Update pinned versions and checksums in code.
  2. Run all static checks and collector tests.
  3. Back up Grafana and collector state.
  4. Deploy central services with docker compose up -d --build.
  5. Apply one runner-host canary.
  6. Validate a real job before applying the remaining hosts.

Never use latest, unverified downloads, or bulk runner restarts.

Uninstall

Remove observability from one Linux host while preserving diagnostics:

sudo bash scripts/runner-observability/uninstall-linux.sh

Use --purge-data only after confirming the exact host and retained log paths.

For Windows:

.\scripts\runner-observability\uninstall-windows.ps1

To remove only the central additions:

cd /opt/homelab/services/monitoring
docker compose stop github-actions-collector alloy
docker compose rm github-actions-collector alloy

Do not remove Prometheus, Loki, Grafana, Alertmanager, or their volumes. Remove the collector volume only after a verified backup and explicit deletion decision.

Troubleshooting

Symptom Check Remediation
Collector not ready docker compose logs github-actions-collector Verify App ID, installation ID, PEM, and App permissions
Repositories missing App installation repository selection Grant the repository to the App; wait for discovery
Runner missing Repository Administration read permission Correct App permission and reinstall/approve changes
Metrics accepted but absent Alloy and Prometheus logs Verify remote-write receiver URL and backend readiness
Logs rejected with 401 Traefik and runner Alloy logs Re-render matching telemetry credentials
Duplicate central logs Both Promtail and Alloy collecting same source Disable that source in one agent during cutover
No runner _diag Host directory ownership and runner unit mount Reapply the runner observability role while idle
GitHub 403 Collector rate-limit metrics and response log Correct permissions or wait for reset; do not retry-loop

Security and limitations

  • Prometheus, Loki, Alloy, cAdvisor, and exporters are never public query endpoints.
  • The telemetry route accepts only authenticated writes; human monitoring UIs remain behind Authentik.
  • The GitHub App is read-only and separate from the administrative runner PAT.
  • Diagnostics can contain repository and error context. Alloy redaction rules drop token-like values, and Loki access inherits Grafana/Authentik controls.
  • Repository, workflow, job, and runner names are bounded by collector guards. High-cardinality execution identifiers are logs, not labels.
  • GitHub polling is eventually consistent; a job shorter than the active poll interval can first appear as a completed event.
  • Organization runner groups apply only when GitHub reports them. Repository-scoped runners display repository scope instead.
  • Native Windows installation is not live-verified until a real Windows runner is provided.