GitHub Actions runner observability¶

This runbook deploys and operates GitHub Actions runner monitoring on the existing infra-services Prometheus, Grafana, Loki, Alertmanager, and Traefik stack. It covers the repository-scoped Linux fleet declared in infra/github-runners/fleet.yml and provides future-ready native Windows onboarding.

Architecture¶

flowchart LR
  GH["GitHub App APIs"] --> Collector["GitHub Actions collector"]
  Collector -->|"Prometheus scrape :9464"| Prometheus
  Linux["Linux runner hosts"] -->|"metrics and logs HTTPS push"| Traefik
  Windows["Future Windows runners"] -->|"metrics and logs HTTPS push"| Traefik
  Traefik --> AlloyCentral["Central Alloy"]
  AlloyCentral --> Prometheus
  AlloyCentral --> Loki
  Prometheus --> Grafana
  Loki --> Grafana
  Prometheus --> Alertmanager
  Loki --> Alertmanager

The collector obtains runner state, workflow runs, and workflow jobs from GitHub for the nine repositories represented in fleet.yml. Host agents scrape exporters locally, then push to the authenticated Alloy gateway. Neither the Hyper-V runner subnet nor exporter ports need to be routable from infra-services.

Prometheus labels contain bounded dimensions such as repository, normalized workflow/job, runner, placement, status, and conclusion. Run IDs, job IDs, commit SHAs, actors, branches, pull requests, URLs, and errors remain structured Loki fields.

GitHub App¶

Create a separate read-only App. Do not reuse the runner-registration PAT.

Open GitHub organization settings → Developer settings → GitHub Apps → New GitHub App.
Set the App name to spadoople-runner-observability.
Disable user authorization and webhook delivery; this collector polls.
Set repository permissions:
Actions: Read-only
Administration: Read-only
Metadata: Read-only (automatic)
Set organization permissions:
Self-hosted runners: Read-only
Create the App, generate a private key, and record the App ID.
Install it on spadoople, selecting every repository that should appear in monitoring.
Record the installation ID from the installation URL.

The App installation defines the maximum collector scope. Production further restricts polling with COLLECTOR_REPOSITORY_INCLUDE_REGEX to the repositories in fleet.yml; update that expression with the manifest when a runner repository is added or retired.

Secrets¶

Copy the example only when the encrypted file does not already exist:

cd /opt/homelab
test -f services/monitoring/.env.sops.yaml ||
  cp services/monitoring/.env.sops.yaml.example \
    services/monitoring/.env.sops.yaml
sops services/monitoring/.env.sops.yaml

Set the existing monitoring values plus:

GITHUB_APP_ID: "123456"
GITHUB_INSTALLATION_ID: "12345678"
GITHUB_ORG: "spadoople"
# Paste the full App private-key PEM via `sops` (prefer a YAML `|` block).
# One-line PEMs are accepted: `render-monitoring-secrets.sh` normalizes them.
# Do not commit plaintext PEMs; the renderer writes a host file readable by the
# collector UID (root:65532 mode 640).
GITHUB_APP_PRIVATE_KEY: "REPLACE_ME_VIA_SOPS"
TELEMETRY_CLIENTS:
  gha_general_a:
    username: "gha-general-a"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  gha_general_b:
    username: "gha-general-b"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  gha_docker_a:
    username: "gha-docker-a"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  gha_docker_b:
    username: "gha-docker-b"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  gha_canary_a:
    username: "gha-canary-a"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  gha_mgmt:
    username: "gha-mgmt"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"
  infra_services:
    username: "infra-services"
    password: "REPLACE_WITH_A_UNIQUE_RANDOM_PASSWORD"

Commit only SOPS ciphertext. Render on infra-services:

cd /opt/homelab
sudo SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt \
  scripts/render-monitoring-secrets.sh
sudo test -s services/monitoring/.env
sudo test -s services/monitoring/secrets/github-app.pem

The renderer uses restrictive temporary files and atomic installation. It creates the Compose environment, the mounted App key, and telemetry client credential files without printing their contents.

Central deployment¶

Confirm telemetry.infra.realemail.app resolves to infra Traefik. The endpoint is machine-authenticated and must not use the browser Authentik middleware.

cd /opt/homelab/services/monitoring
docker compose --env-file .env config --quiet
cd /opt/homelab
bash scripts/validate-monitoring.sh

Back up before changing containers:

sudo restic backup \
  /var/lib/docker/volumes/monitoring_grafana-data/_data \
  /var/lib/docker/volumes/monitoring_collector-data/_data

Start the new services:

cd /opt/homelab/services/monitoring
docker compose --env-file .env up -d --build github-actions-collector alloy
docker compose ps github-actions-collector alloy prometheus loki grafana alertmanager
docker compose logs --tail=100 github-actions-collector alloy

Recreate backend services only after configuration validation:

docker compose --env-file .env up -d prometheus loki grafana alertmanager

Central Docker and syslog shipping is part of monitoring/alloy/config.alloy (Promtail removed 2026-07-27). After Alloy config changes, validate and recreate:

docker run --rm -v /opt/homelab/monitoring/alloy:/etc/alloy:ro \
  grafana/alloy:v1.21.0 validate /etc/alloy/config.alloy
docker compose --env-file .env up -d alloy

Linux runner rollout¶

The authoritative path is Ansible. Schedule this while the selected runner host is idle because adding persistent diagnostic mounts restarts its runner units.

Canary one host:

cd /opt/homelab
ANSIBLE_CONFIG=infra/ansible/ansible.cfg ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml \
  --limit gha-general-a

Verify, then deploy the remaining hosts:

cd /opt/homelab
ANSIBLE_CONFIG=infra/ansible/ansible.cfg ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml \
  --limit 'gha-general-b:gha-docker-a:gha-docker-b:infra-services'

Hyper-V NAT VMs use public DNS (1.1.1.1) and cannot resolve *.infra.realemail.app via AdGuard. The observability role pins telemetry.infra.realemail.app → 192.168.6.17 in /etc/hosts by default (github_runner_observability_pin_telemetry_hosts). Disable that pin on LAN hosts that already resolve through AdGuard (see infra-services in github-runners.yml). The role also applies named + default ACLs so Alloy can read runner _diag logs written as uid 10001.

For a Linux runner outside Ansible:

sudo bash scripts/runner-observability/install-linux.sh \
  --telemetry-url https://telemetry.infra.realemail.app \
  --username gha-general-a \
  --password-file /secure/path/host-telemetry-password

Only outbound TCP 443 is required. Do not expose node_exporter, cAdvisor, or Alloy listeners to untrusted networks.

Windows runner onboarding¶

No native Windows runner is currently designated, so this path is statically validated but not considered live-verified.

From an elevated PowerShell 7 session:

Set-Location C:\path\to\homelab
.\scripts\runner-observability\install-windows.ps1 `
  -TelemetryUrl 'https://telemetry.infra.realemail.app' `
  -TelemetryUsername 'windows-runner-host' `
  -TelemetryPasswordFile 'C:\secure\telemetry-password'

The script installs windows_exporter and Alloy as Windows services, validates download hashes, discovers runner services and _diag directories, and creates only outbound telemetry configuration.

Verification¶

Offline/static checks:

cd /opt/homelab
bash scripts/validate-monitoring.sh

Live checks after the central deployment:

cd /opt/homelab
bash scripts/validate-monitoring.sh --live
docker exec github-actions-collector \
  /github-actions-collector healthcheck http://localhost:9464/readyz
docker exec prometheus wget -qO- \
  'http://github-actions-collector:9464/metrics' |
  grep '^github_actions_runner_online'

After the owner authorizes a notification test, submit the synthetic alert and confirm it reaches the designated test destination:

bash scripts/validate-monitoring.sh --live --synthetic-alert

Run a safe workflow and verify:

The job appears queued on Current GitHub Actions Activity.
runner becomes populated when GitHub assigns it.
Queue and execution durations appear once after completion.
The host dashboard shows the corresponding CPU, memory, disk, network, and runner-container interval.
Runner journal and _diag entries appear in the log dashboard for the same runner and time range.

Dashboards and alerts¶

Grafana provisions six dashboards in the existing Homelab folder:

Runner Fleet Overview
Current GitHub Actions Activity
Historical Activity and Utilization
GitHub Actions CI Health
Runner Host Health
Runner Logs and Diagnostics

Prometheus alerts cover runner availability (LAN / always-on only for offline and Hyper-V host-down; on-demand guests including canary-a sleeping is expected), sustained demand without idle capacity, queue age, queue percentiles, failure rates, inactivity, API health, rate-limit pressure, exporter/collector ingestion, storage, certificate, CPU, memory, disk, latency, and OOM conditions.

Loki rules cover repeated reconnects, runner update failures, and worker exceptions. Existing Discord and ntfy routes remain enabled. Optional Email, Slack, and webhook examples are under monitoring/alertmanager/examples/.

Backup and restore¶

Grafana and collector state are backed up. Prometheus and Loki remain retention-bounded and regeneratable.

sudo restic backup \
  /var/lib/docker/volumes/monitoring_grafana-data/_data \
  /var/lib/docker/volumes/monitoring_collector-data/_data

Stage a reviewed snapshot before any stop, deletion or live copy. Use a root Bash shell on the Linux recovery host, Python 3 with PyYAML (python3-yaml), restic on PATH, and the reviewed checkout at /opt/homelab. /tmp/restore-monitoring must be absent, with existing non-symlink parents. Do not pre-create it or reuse a previous attempt.

umask 077
export RESTIC_REPOSITORY=/mnt/infra-backups/homelab-restic/monitoring
export RESTIC_PASSWORD_FILE=/etc/restic/password
restic snapshots --json || exit 1
read -r -p "Reviewed complete-path snapshot ID: " SNAPSHOT_ID
python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/monitoring/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-monitoring || exit 1

Review the default metadata/preflight JSON for the host, time, ID and both roots. Then explicitly stage into the new private mode-0700 target:

python3 /opt/homelab/backups/restore-snapshot.py \
  --manifest /opt/homelab/services/monitoring/backup.yml \
  --snapshot "$SNAPSHOT_ID" --target /tmp/restore-monitoring --restore || exit 1

The producer repair in PR #469 uses one invocation for all manifest paths with tag homelab-path-set-v1. It does not make live SQLite data application-consistent. This is source work, not a deployment or drill claim. Old snapshots remain split: select reviewed disjoint same-host IDs with repeated --snapshot arguments, exactly covering both roots, as described in Restore. --latest is scheduled verification only; the newest snapshot must be complete, with no fallback or automatic mixing.

Validate all staging and both SQLite databases in isolation before stopping either service or copying anything into live volumes. Path completeness does not prove a valid backup; remaining application-specific recovery defects are not certified or fixed by this repair. Failed staging stays in place: preserve evidence and choose a new absent target for any retry. The helper does not promote or clean up.

After validation and separate owner approval, review the existing volume procedure in Restore. Only after that application cutover:

cd /opt/homelab/services/monitoring
docker compose --env-file .env up -d grafana github-actions-collector
docker compose logs --tail=100 grafana github-actions-collector

Loss of collector SQLite does not lose GitHub data, but the next lookback can re-observe completed events and temporarily distort counters. Restore it when available.

Upgrade¶

Update pinned versions and checksums in code.
Run all static checks and collector tests.
Back up Grafana and collector state.
Deploy central services with docker compose up -d --build.
Apply one runner-host canary.
Validate a real job before applying the remaining hosts.

Never use latest, unverified downloads, or bulk runner restarts.

Uninstall¶

Remove observability from one Linux host while preserving diagnostics:

sudo bash scripts/runner-observability/uninstall-linux.sh

Use --purge-data only after confirming the exact host and retained log paths.

For Windows:

.\scripts\runner-observability\uninstall-windows.ps1

To remove only the central additions:

cd /opt/homelab/services/monitoring
docker compose stop github-actions-collector alloy
docker compose rm github-actions-collector alloy

Do not remove Prometheus, Loki, Grafana, Alertmanager, or their volumes. Remove the collector volume only after a verified backup and explicit deletion decision.

Troubleshooting¶

Symptom	Check	Remediation
Collector not ready	`docker compose logs github-actions-collector`	Verify App ID, installation ID, PEM, and App permissions
Repositories missing	App installation repository selection	Grant the repository to the App; wait for discovery
Runner missing	Repository Administration read permission	Correct App permission and reinstall/approve changes
Metrics accepted but absent	Alloy and Prometheus logs	Verify remote-write receiver URL and backend readiness
Logs rejected with 401	Traefik and runner Alloy logs	Re-render matching telemetry credentials
Duplicate central logs	Accidental re-enable of retired Promtail config	Ensure only Alloy ships Docker/syslog
No runner `_diag`	Host directory ownership and runner unit mount	Reapply the runner observability role while idle
GitHub 403	Collector rate-limit metrics and response log	Correct permissions or wait for reset; do not retry-loop
Installation budget repeatedly reaches zero	Repository scope or polling intervals are too broad	Keep active polling at two minutes, history at one hour, and limit collection to fleet repositories

Security and limitations¶

Prometheus, Loki, Alloy, cAdvisor, and exporters are never public query endpoints.
The telemetry route accepts only authenticated writes; human monitoring UIs remain behind Authentik.
The GitHub App is read-only and separate from the administrative runner PAT.
Diagnostics can contain repository and error context. Alloy redaction rules drop token-like values, and Loki access inherits Grafana/Authentik controls.
Repository, workflow, job, and runner names are bounded by collector guards. High-cardinality execution identifiers are logs, not labels.
GitHub polling is eventually consistent; a job shorter than the active poll interval can first appear as a completed event.
Active state is polled every two minutes and completed history every hour. Processed completed runs are skipped before job retrieval so API cost does not grow with the 24-hour lookback window.
Organization runner groups apply only when GitHub reports them. Repository-scoped runners display repository scope instead.
runner_group comes from GitHub and exists only on job-level series. runner_class comes from fleet.yml and is the label carried by github_actions_runner_info, _online, and _busy. Fleet dashboards filter runners by runner_class.
github_actions_last_job_completion_timestamp_seconds and github_actions_last_success_timestamp_seconds carry only org, repository, workflow, job_name, and runner. Adding scope, placement_group, runner_class, or os to a query against them silently returns no series.
github_actions_oldest_queued_job_timestamp_seconds carries only org, repository, workflow, and job_name. It has no runner label, because a queued job has not been assigned to a runner yet.
Both last-timestamp metrics are keyed per workflow and job. Aggregate with max by (runner) before treating a value as "time since this runner last did anything", or a runner looks inactive whenever any one of its workflows has been idle. GitHubRunnerInactive uses that aggregation and a thirty-day gate so quiet Dependabot-only stretches do not ticket; investigate only when a registration looks abandoned.
github_actions_last_success_timestamp_seconds does not exist at all for a workflow that has never succeeded within the retained history, so a query for stale successes cannot see never-green workflows. CI Health surfaces those separately with an unless against github_actions_jobs_total.
Native Windows installation is not live-verified until a real Windows runner is provided.