Central GitHub runner management¶

This is the authoritative runbook for observing, qualifying, and maintaining the GitHub Actions runner fleet. The central dispatcher is .github/workflows/runner-fleet-dispatch.yml; local reusable workflows perform observe (_runner-fleet-observe.yml), qualify (_runner-fleet-qualify.yml), watchdog (_runner-fleet-watchdog.yml), and maintenance (_runner-fleet-maintain.yml) operations. The scheduled workflow runs observation every six hours and the stale-run watchdog every 30 minutes.

API discovery, mutation preview, observe, and watchdog planning run on the always-on LAN runner (RUNNER_FLEET_API_RUNS_ON, falling back to DEPLOY_RUNS_ON / homelab-lan-01 on infra-services). Qualification jobs still run on the selected workers. Protected apply and fresh-state recheck use the management runner (RUNNER_FLEET_MAINTENANCE_RUNS_ON, label runner-management on gha-mgmt / homelab-mgmt-01), which the Hyper-V scaler starts when those jobs queue. This keeps Ansible to Hyper-V NAT off the LAN box, without paying for GitHub-hosted minutes.

Zero-hosted policy: repository workflows must not use ubuntu-latest, windows-latest, or macos-latest. Enforce with scripts/check-no-hosted-runners.sh (Lint job No GitHub-hosted runners). Use CI_RUNS_ON, DOCKER_RUNS_ON, DEPLOY_RUNS_ON, RUNNER_FLEET_API_RUNS_ON, or RUNNER_FLEET_MAINTENANCE_RUNS_ON instead.

Qualify topology limit¶

Repository-scoped runners only accept jobs from their own repository. The central homelab qualify workflow therefore fails closed when the selected targets belong to another repository (for example tiktooker or dnd_session_parser). Use observe / health / drift from homelab for cross-repo visibility, and dispatch qualify from the target repository (or limit qualify scope to homelab runners) until per-repo qualify receivers exist.

flowchart LR
  Operator["Owner / protected approval"] --> Dispatch["Central dispatcher"]
  Schedule["Scheduled workflow"] --> Observe["Observe / watchdog preview"]
  Dispatch --> Observe
  Dispatch --> Qualify["Exact-runner qualification"]
  Dispatch --> Plan["Maintenance preview"]
  Observe --> API["Management runner control plane"]
  Qualify --> Fleet["Selected runner"]
  Plan --> Approval["runner-fleet-maintenance approval"]
  Approval --> Manager["Ansible management runner"]
  Manager --> Target["Exact non-management target"]

Fleet and trust boundaries¶

The fleet contains thirty-two repository-scoped Linux/X64 runner instances (including homelab-docker-01 and control-plane homelab-mgmt-01). General workers are ephemeral unprivileged containers. Docker workers are placed on isolated build VMs and are the only class with the Docker socket. The homelab LAN worker has host networking, a read-only operational configuration mount, and a read-write mount of the live /opt/homelab checkout. It never receives dev or dependabot labels. The management instance carries runner-management and must not carry homelab,ci (so ordinary Lint never schedules there). Applied fleet maintenance must never target homelab-mgmt-01 or host gha-mgmt.

No self-hosted runner receives the special dependabot label. GitHub-generated Dependabot update jobs require Docker access, so they remain on GitHub-hosted infrastructure. CI for Dependabot pull requests still uses the repository's ordinary ci runners. Keep Settings > Advanced Security > Dependabot on self-hosted runners disabled in every repository.

Placement¶

The four CI/build placement groups run as on-demand Hyper-V VMs on CAPTAINKANGAPOO. The Windows host stays up; the guests Stop-VM after about 15 minutes idle and start when a matching job queues. First job after sleep typically waits 1–3 minutes for boot, Docker, and the listener. The LAN runner remains always-on on infra-services (placement:lan-a, inventory ansible_user: someone; the fleet apply key must be authorized there so homelab-mgmt-01 can reach it when that guest is awake). The management control-plane VM is gha-mgmt (see Create gha-mgmt Hyper-V guest); it sleeps with the rest of the fleet and wakes for qualify / maintain apply.

Current Hyper-V dynamic-memory targets (after the 2026-08-05 capacity grow) still apply while a guest is running:

VM	Startup	Min	Max
`gha-general-a` / `gha-general-b`	12 GiB	6 GiB	28 GiB
`gha-docker-a` / `gha-docker-b`	10 GiB	4 GiB	16 GiB
`gha-mgmt`	4 GiB	2 GiB	8 GiB

Raising startup/minimum requires a brief VM stop; maximum can change while running. Host-specific github_runner_instances in infra/ansible/inventory/github-runners.yml contain only the entries assigned to each host. Placement → VM names live in infra/github-runners/hyperv-scaler/placements.yml.

Homelab Docker CI (DOCKER_RUNS_ON) uses instance homelab-docker-01 on gha-docker-b — an additional ephemeral docker-class worker, not a new Hyper-V guest.

Telegram Docker CI uses telegram-docker-01 on gha-docker-b. Its repository variable must be DOCKER_RUNS_ON=["self-hosted","Linux","X64","Telegram","docker"]; keep ordinary lint and test jobs on CI_RUNS_ON so only service-container work crosses the Docker socket boundary.

On-demand Hyper-V scaler¶

GitHub only assigns work to online runners. A loop on the Hyper-V host polls queued/in-progress jobs (~15s), starts the placement VMs that can take them, and ACPI-stops guests after idle_timeout_seconds (900) with no needed or busy runners.

Stays up	Sleeps
Windows workstation, scaler task, `homelab-lan-01`	`gha-general-a/b`, `gha-docker-a/b`, `gha-mgmt`

Do not Stop-VM the fleet until the scaler has woken a guest for a real queued workflow. Otherwise jobs sit in queue until GitHub times them out.

Owner install on CAPTAINKANGAPOO (PowerShell 7, Hyper-V Administrators, uv on PATH):

Copy the fleet read App PEM to C:\ProgramData\homelab\hyperv-scaler\fleet-read.pem (icacls for your user only), or use gh auth token from config.ps1 (live path).
Edit C:\ProgramData\homelab\hyperv-scaler\config.ps1 with HYPERV_SCALER_GITHUB_APP_ID, HYPERV_SCALER_GITHUB_APP_PRIVATE_KEY_FILE, and HYPERV_SCALER_GITHUB_APP_INSTALLATION_ID (or a GH_TOKEN with Actions read on every fleet repository).
From an elevated pwsh in the repo:

.\infra\github-runners\hyperv-scaler\Install-HyperVRunnerScaler.ps1

That registers HomelabHyperVRunnerScaler for startup and current-user logon, starts a missed trigger when the user session becomes available, and removes the Scheduled Task's default 72-hour execution limit. It also sets AutomaticStartAction=Nothing on the five guests, but does not power them off. 4. Dry-run: $env:HYPERV_SCALER_DRY_RUN='1' then .\infra\github-runners\hyperv-scaler\Invoke-HyperVRunnerScaler.ps1 -Once. 5. Prove a wake with one stopped guest and a matching queued job, then allow idle stop for the rest of the fleet.

Live (2026-08-21)¶

Task HomelabHyperVRunnerScaler is running as CAPTAINKANGAPOO\renjamin. All five guests have AutomaticStartAction=Nothing. Auth is gh auth token via C:\ProgramData\homelab\hyperv-scaler\config.ps1 (not a committed PEM). The task must remain Running; Ready with queued fleet work means the current-user scaler missed startup or exited.

Each tick failure is appended to C:\ProgramData\homelab\hyperv-scaler\scaler.log and the scaler continues polling. The scheduled task is configured to restart unexpected process exits up to 999 times at one-minute intervals. After installing an updated scaler, verify the task remains Running and inspect this log before manually starting a stopped fleet guest.

The installed task launches PowerShell with -WindowStyle Hidden. Do not run the long-lived scaler manually in an interactive terminal: closing that window or pressing Ctrl+C terminates the process and leaves the task Ready until it is restarted.

Because the task uses the operator's interactive token, the startup trigger cannot launch it before that user session exists. The separate logon trigger is therefore required after every reboot. Verify both triggers after installation:

(Get-ScheduledTask -TaskName HomelabHyperVRunnerScaler).Triggers |
    Select-Object CimClass

The output must contain both MSFT_TaskBootTrigger and MSFT_TaskLogonTrigger.

Wake proof: gha-mgmt was stopped, a qualify dispatch queued runner-management, the scaler started the guest (~1 minute Off→Running), and run 32547732271 succeeded. Repository variable RUNNER_FLEET_API_RUNS_ON is set to the same JSON as DEPLOY_RUNS_ON.

Ansible to 172.29.240.11–.15 still only works from CAPTAINKANGAPOO / gha-mgmt (Ansible from CAPTAINKANGAPOO). Keep maintain apply on RUNNER_FLEET_MAINTENANCE_RUNS_ON.

Idle Hyper-V runners looking offline in GitHub is expected. Prometheus GitHubRunnerOffline / GitHubRunnerServiceStopped / Hyper-V NodeExporterUnavailable / RunnerTelemetryUnavailable ignore those placements. GitHubActionsQueueLong and GitHubRunnerNoIdleCapacity (10 minutes) remain the scaler-failure signal.

The canonical control-plane manifest is infra/github-runners/fleet.yml. It defines expected identity, repository, labels, placement group, pinned runner version, adapter contract, thresholds, and explicit deregistration prohibition. Ansible inventory continues to define the physical host mapping. A runner name must map to exactly one inventory host before maintenance can run.

Current topology assumptions:

Every configured instance is repository-scoped, linux-x64, and uses the ansible@v1 maintenance adapter.
Repository-scoped runners cannot belong to GitHub organization runner groups. Use placement scope for this fleet's general-a, docker-b, and similar topology groups. group is reserved for an explicitly declared organization runner group.
Organization-scoped runners and groups are supported by the API model only when explicitly represented in a future manifest. They require organization permissions and must not be inferred from repository runners.
The Linux reference host adapter manages systemd units and Docker containers. A Windows local-service adapter defines the same bounded v1 contract, but no Windows runner is currently present in the canonical manifest. Windows reporting and qualification are ready; applied Windows maintenance requires a reviewed transport adapter that invokes the local PowerShell contract with exact service/install identities. The built-in ansible@v1 registry rejects that topology. macOS and ARM are selection vocabulary, not deployed topology.

Registration credential¶

Create infra/ansible/inventory/group_vars/github_runner_hosts/registration.sops.yaml from the adjacent example and encrypt it with SOPS before committing or deploying. The value is never a workflow secret and must not appear in normal inventory, logs, or shell history. Use a fine-grained PAT restricted to runner administration on every repository that appears in infra/github-runners/fleet.yml. When adding a new repo's instances, extend that PAT's repository allowlist before expecting registration to succeed. A missing repo usually returns HTTP 404 (not 403) from /repos/{owner}/{repo}/actions/runners/registration-token, and the unit crash-loops with curl: (22) The requested URL returned error: 404.

This host registration PAT remains inside the SOPS/host boundary. Do not copy it into GitHub Actions. Central management uses a dedicated GitHub App with short-lived installation tokens; it does not replace the host credential used to obtain runner registration tokens.

Configure the control plane¶

Dedicated GitHub Apps¶

Use separate Apps so a read-only or unattended workflow never receives a private key capable of broader mutation:

Runner Fleet Read — Actions read, repository Administration read, organization Self-hosted runners read, and Metadata read.
Runner Fleet Operator — repository Administration write, organization Self-hosted runners write, and Metadata read. This key exists only in the reviewer-protected maintenance environment. Write access removes scheduling labels before drain; there is no runner-delete operation.
Runner Fleet Watchdog — Actions write and Metadata read only. This key can force-cancel stale runs but cannot administer runner registrations.

Install each App only on managed repositories. Organization Self-hosted runners permission is needed only when organization-scoped runners/groups enter the manifest. Repository-scoped runners cannot become members of those groups.

Configure:

Keep Issues enabled in the control repository so scheduled alerts can reconcile the runner-fleet-alert issue with GITHUB_TOKEN.
Repository variables RUNNER_FLEET_READ_CLIENT_ID, RUNNER_FLEET_OPERATOR_CLIENT_ID, and RUNNER_FLEET_WATCHDOG_CLIENT_ID (GitHub App Client ID, Iv1.…). Fleet workflows pass these as client-id to actions/create-github-app-token. Do not leave them empty or token mint fails. Numeric RUNNER_FLEET_*_APP_ID vars may still exist for older tooling (Hyper-V scaler uses App ID locally); they are not used by fleet workflows. The spadoople-runner-observability App is SOPS on infra-services, not a repo variable.
RUNNER_FLEET_READ_APP_PRIVATE_KEY only in runner-fleet-readonly.
RUNNER_FLEET_OPERATOR_APP_PRIVATE_KEY only in runner-fleet-maintenance.
RUNNER_FLEET_WATCHDOG_APP_PRIVATE_KEY in runner-fleet-maintenance and runner-fleet-automation.
Repository variable RUNNER_FLEET_API_RUNS_ON as a JSON runner-label array matching DEPLOY_RUNS_ON (["self-hosted","Linux","X64","homelab","lan","deploy"]). Observe, watchdog, dispatcher validate, and the scheduled alert job use this (with fallback to DEPLOY_RUNS_ON if unset).
Repository variable RUNNER_FLEET_MAINTENANCE_RUNS_ON as a JSON runner-label array: ["self-hosted","Linux","X64","runner-management"]. Required for qualify and maintain apply — there is no ubuntu-latest fallback. Guest gha-mgmt and instance homelab-mgmt-01 are live (2026-08-09, 172.29.240.15) and on-demand after the scaler is installed.
Repository variable RUNNER_FLEET_ANSIBLE_PRIVATE_KEY_FILE — absolute path inside the homelab-mgmt-01 container to an SSH private key that can log in as runneradmin on the other Hyper-V runner guests (same key as ~/.ssh/github-runner-vm-admin on CAPTAINKANGAPOO). Live path: /etc/homelab/runner-fleet/id_ed25519 (host file, ACL’d for runner UID 10001, mounted by launch-runner for management class). Do not point this at /home/runneradmin/.ssh/... — fleet jobs run as container uid 10001 and cannot see the host home directory (no such identity / Permission denied (publickey)).
Fleet apply sets ANSIBLE_CONFIG=infra/ansible/ansible-fleet.cfg (host-key checking off, roles_path, no ara). The adapter cwd is the repo root, so infra/ansible/ansible.cfg is not auto-loaded; without the fleet cfg, apply fails with Host key verification failed even when the key works.
Optional repository variable for adapter timeout overrides as documented elsewhere.

Keep all private keys in environment secrets rather than repository-level secrets. Rotate one App at a time by adding its new key, updating only its listed environments, verifying the affected operation, and deleting the old key.

Protected environments¶

Create these GitHub environments:

runner-fleet-readonly for API discovery, matrix construction, and plan generation. Restrict deployment branches to protected main; do not expose the read App key to feature-branch workflow code.
runner-fleet-maintenance for operator-requested host mutation. Require the owner as reviewer, prevent self-review where available, and restrict deployment branches to the protected default branch.
runner-fleet-automation for the existing scheduled stale-run watchdog. Do not add a required reviewer, because that would leave a deployment waiting every 30 minutes. Restrict it to the default branch and store the Actions-only watchdog key.

Environment approval is a required apply gate, not a substitute for the request's authorization fields. Owner approval is required for live disruptive maintenance. Scheduled cancellation is explicit policy in the scheduled workflow and still uses preview, age thresholds, fingerprints, and a fresh per-run recheck.

The dispatcher and scheduler use secrets: inherit when calling the local reusable workflows. GitHub otherwise resolves secrets from the reusable job's environment as empty. Keep these callers and reusable workflows in this repository on protected main; inheritance also makes caller-visible repository and organization secrets available to the called workflow.

Management runner prerequisites¶

The runner selected by RUNNER_FLEET_MAINTENANCE_RUNS_ON must be a trusted, dedicated management path with:

a clean checkout of the default branch and access to Python package installation; the workflow pins Python, PyYAML, and ansible-core;
both generated and GitHub-runner Ansible inventories;
the maintenance playbook and installed ansible@v1 adapter;
SSH and sudo reachability to the exact target host, without exposing its key to ordinary CI runners;
RUNNER_FLEET_ADAPTER_MODE=execute and RUNNER_FLEET_CONTROL_RUNNER_NAME set to its exact runner name;
the Hyper-V scaler able to start gha-mgmt when a runner-management job queues (the guest is not 24/7 once on-demand is live).

Homelab maintenance must never target the management runner executing the operation. The adapter rejects self-maintenance, and operators must choose a repository, placement-group, or exact-runner scope that excludes it. Do not run fleet-wide applied maintenance until a second management runner provides a non-self target path.

Create gha-mgmt Hyper-V guest¶

Status (2026-08-09): Guest exists and is online — gha-mgmt at 172.29.240.15, runner homelab-mgmt-01 listening with runner-management. Steps below are historical recreate notes only.

Repo automation installs runners on existing Linux guests only. Creating gha-mgmt is an owner Hyper-V task on CAPTAINKANGAPOO (Administrator PowerShell). Prefer cloning a known-good sibling rather than a bare ISO.

0. Inspect siblings (do this first)¶

Get-VM -Name gha-docker-b | Format-List Name, State, Generation, Path, Status
Get-VMNetworkAdapter -VMName gha-docker-b | Format-List *
Get-VMMemory -VMName gha-docker-b | Format-List *
Get-VMHardDiskDrive -VMName gha-docker-b | Format-List *
Get-VMSwitch | Format-Table Name, SwitchType

Note the vSwitch name, Generation, and VHDX path. Inside the guest (SSH as runneradmin@172.29.240.14) also record:

ip -br a
ip route
cat /etc/resolv.conf
hostnamectl

Use the same prefix/gateway/DNS for gha-mgmt. Planned address: 172.29.240.15 (confirm unused).

1. Preferred: clone gha-docker-b¶

Shut down or live-export gha-docker-b only if required by your Hyper-V version; otherwise use Export-VM / Import-VM / Copy-VM as available on your host.
Create VM name gha-mgmt from that export/copy on the same vSwitch.
Set dynamic memory: Startup 4 GiB, Minimum 2 GiB, Maximum 8 GiB.
First boot: change hostname to gha-mgmt, set static IP .15, regenerate machine-id and SSH host keys (sudo rm -f /etc/ssh/ssh_host_* then sudo dpkg-reconfigure openssh-server or equivalent), clear any old github-runner@* units that still point at cloned instance names.
Confirm SSH: ssh runneradmin@172.29.240.15.
Ensure /etc/homelab/age-key.txt exists if siblings use SOPS on-host (copy from a sibling only via your normal secret path — do not commit it).

2. Alternate: Ubuntu 24.04 Gen2 from ISO¶

Only if you have no clone path: Gen2 VM, same vSwitch, Ubuntu Server 24.04, user runneradmin with SSH key auth + passwordless sudo, install Docker CE matching siblings, place age key, static IP .15.

3. Register via Ansible (after inventory is on main / this branch)¶

Inventory host gha-mgmt and instance homelab-mgmt-01 are already declared in infra/ansible/inventory/github-runners.yml and infra/github-runners/fleet.yml. Labels must include runner-management and must not include homelab or ci.

ANSIBLE_CONFIG=infra/ansible/ansible.cfg ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml \
  --limit gha-mgmt

4. GitHub variable¶

Repo Settings → Secrets and variables → Actions → Variables:

RUNNER_FLEET_MAINTENANCE_RUNS_ON = ["self-hosted","Linux","X64","runner-management"]

Confirm homelab-mgmt-01 is online with those labels, then re-run any queued fleet workflows.

Thresholds and overrides¶

The manifest defaults are:

Setting	Default	Actions variable / environment override
In-progress run stale age	360 minutes	`RUNNER_FLEET_STALE_RUN_MINUTES`
Queued run stale age	30 minutes	`RUNNER_FLEET_STALE_QUEUE_MINUTES`
Observation activity window	24 hours	`RUNNER_FLEET_RECENT_RUN_HOURS`
Maximum selected targets	32	`RUNNER_FLEET_TARGET_CAP`
GitHub API retry attempts	4	`RUNNER_FLEET_RETRY_ATTEMPTS`

RUNNER_FLEET_PLAN_TTL_MINUTES defaults to 30, and request parallelism is bounded by RUNNER_FLEET_MAX_PARALLEL (default 8). Prefer changing the canonical manifest through review. RUNNER_FLEET_ADAPTER_TIMEOUT_SECONDS defaults to 1800 for host work. Use overrides only for a time-bounded incident, record the reason, and restore defaults afterward.

Deploy¶

Build the isolated general and Docker VMs and confirm the LAN host is reachable.
Review infra/ansible/inventory/github-runners.yml and confirm its host addresses and instance allocation match the deployed fleet.
Create and SOPS-encrypt infra/ansible/inventory/group_vars/github_runner_hosts/registration.sops.yaml from the adjacent example. Commit only the ciphertext.
Run the playbook with both inventory sources (see Ansible from CAPTAINKANGAPOO when operating from the Hyper-V host):

ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml

For a single new instance, limit to its host and pass an allowlist that contains only names present on that host (the role asserts allowlist ⊆ host instances):

ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml \
  --limit gha-docker-b \
  -e '{"github_runner_instance_allowlist":["rexit-evolved-docker-01"]}'

Set repository variables:
CI_RUNS_ON=["self-hosted","Linux","X64","<repo>","ci"]
DOCKER_RUNS_ON=["self-hosted","Linux","X64","<repo>","docker"]
DEPLOY_RUNS_ON=["self-hosted","Linux","X64","homelab","lan","deploy"]
Dispatch runner qualification before moving CI workloads.

The LAN container runs as lanrunner UID/GID 1001, matching the owner of the live checkout and Komodo credential source. Its launcher mounts /opt/homelab read-write, reapplies a read-only ACL for UID 1001 to /etc/homelab/labctl.env, and deliberately fails before registration if the checkout ownership or Komodo credential readability is wrong. Do not make either credential source world-readable. General and Docker runners continue to run as UID/GID 10001 and never receive the live checkout mount.

The pinned runner image includes gnupg because maintained actions such as Codecov validate downloaded helper binaries with GPG. Treat missing action runtime dependencies as an image contract failure, not a reason to disable the action's integrity validation.

The pinned runner image includes libssl-dev and pkg-config because Rust crates that link OpenSSL natively (for example openssl-sys, pulled in transitively by reqwest's default native-tls backend) fail their build script without system OpenSSL headers, matching ubuntu-latest's toolchain.

The runner installation and job workspace under /runner use the container's ephemeral disk-backed writable layer. Do not mount /runner as tmpfs: package installs, checkouts, and build artifacts can otherwise exhaust the runner's memory cgroup even when process RSS is modest. The container is launched with --rm, so its writable layer is discarded after every job.

Central dispatcher operations¶

Open Actions → Runner fleet operations → Run workflow on the protected default branch (main). Every request is normalized before routing, and selectors resolve only against infra/github-runners/fleet.yml.

You do not SSH into Hyper-V VMs for normal revive, cancel, or maintenance. The dispatcher is the operator UI; SSH is last-resort when the Ansible adapter or management runner itself is broken.

Form fields¶

Field	What to put
`operation`	One of the operations in the table below
`fleet_target`	Preferred. Dropdown generated from `infra/github-runners/fleet.yml` (`placement:…`, `repository:…`, `runner:…`). Overrides `scope_type` / `scope_value` unless you leave the manual sentinel
`scope_type`	Manual fallback when `fleet_target` is the sentinel
`scope_value`	Manual fallback; empty for `fleet`
`platform`	Leave `all` unless isolating Linux vs Windows
`dry_run`	true for preview (default). false only when applying
`authorize_disruptive`	false for preview. true only with apply + a real reason
`drain_mode`	`graceful` unless owner explicitly authorizes `immediate`
`target_version`	Only for `upgrade`; must equal the manifest pin
`max_parallel`	Keep `1` until a canary succeeds
`reason`	Required for apply: 10–500 characters (incident or change note)

GitHub Actions cannot call the inventory API when rendering workflow_dispatch forms. Choices are therefore generated into the workflow YAML from fleet.yml by:

uv run python inventory/generators/render-runner-fleet-dispatch.py

CI Generator drift check fails if the dropdown or diagram drifts. After editing fleet.yml, regenerate and commit both outputs (pre-commit hook runner-fleet-generators also rewrites them when fleet.yml is staged):

uv run python inventory/generators/render-runner-fleet-dispatch.py
uv run python inventory/generators/render-runner-fleet-diagram.py

Placement map: runner-fleet-diagram.md.

Scope examples¶

Goal	Prefer `fleet_target`	Manual fallback
Whole fleet (read-only or watchdog preview)	sentinel + `scope_type=fleet`	`fleet` / empty
One repo's runners	`repository:dnd_session_parser`	`repository` / `dnd_session_parser`
One Hyper-V placement	`placement:general-a`	`placement` / `general-a`
Exact instance	`runner:dnd-session-parser-docker-01`	`runner` / exact name
Label intersection	sentinel + `labels`	`labels` / `docker` or `homelab,ci`

Exact runner names are case-sensitive and must match fleet.yml (for example tiktooker-docker-02, not Tiktooker-Docker-02).

Operations reference¶

Operation	Purpose	Mutation behavior
`inventory`	List expected and live runner identity and availability	Read-only
`health`	Report missing, offline, busy, and unexpected runners	Read-only
`activity`	Summarize recent run and job outcomes	Read-only
`queue`	Report queue pressure and oldest queued job age	Read-only
`report`	Produce the complete inventory, health, drift, queue, and activity report	Read-only
`drift`	Compare labels, platform, group, and manifest state	Read-only
`qualify`	Run identity, pinned-version, toolchain, temporary-file, and free-space checks on each exact selected runner	Workload probe only
`watchdog`	Find stale queued or in-progress workflow runs and optionally force-cancel them	Preview first; guarded API mutation
`maintenance`	Restore manifest-declared local service state where drift exists	Guarded host mutation
`cleanup`	Remove only an exact drained runner's container and diagnostics	Guarded host mutation
`upgrade`	Verify and apply the manifest-pinned runner version	Guarded host mutation
`restart`	Restart an exact drained and idle runner service	Guarded host mutation
`drain`	Stop new work and wait a bounded time for current work, or use explicitly authorized immediate mode	Guarded host mutation
`resume`	Remove the drain marker and resume the exact service	Guarded host mutation
`remediate`	Reconcile one exact drifted/offline runner; otherwise escalate	Guarded exact-target mutation

upgrade additionally requires target_version equal to the manifest-pinned version. Start with max_parallel: 1.

Run drain before maintenance, cleanup, restart, upgrade, or remediate; those operations fail closed without the exact drain marker. Draining an already-offline target is valid: it establishes the marker without interrupting work, after which exact-target remediation can reconcile the service. Finish with resume only after verification.

How to use matrix¶

What you see	First do (always dry-run)	Then apply when preview looks right	Do not
After a GitHub Actions outage: jobs stuck queued / in_progress	`watchdog` · `scope_type=fleet` · `dry_run=true`	Same scope · `dry_run=false` · `authorize_disruptive=true` · reason about the outage	Cancel healthy work while GitHub status is still red
Grafana shows offline / missing runners	`health` or `report` · `fleet` or one `repository`	See Revive dead runners below	SSH every VM first
Runner online in GitHub but never picks jobs (zombie session)	`health` + `drift` on that exact `runner`	`drain` → `remediate` (or `restart`) → `resume` on that exact name	Bulk `fleet` restart
Labels / class wrong vs `fleet.yml`	`drift` on `repository` or `runner`	`drain` → `remediate` → `resume`	Hand-edit labels in the GitHub UI
Local systemd/image drift suspected	`maintenance` dry-run on `placement` or `runner`	`drain` each target → apply `maintenance` → `resume`	Apply `maintenance` with `fleet` scope
Need confidence a worker is healthy	`qualify` on one exact `homelab` runner first	Expand place/repo only after canary passes	Cross-repo `qualify` from `homelab` (fails closed)
Disk / toolchain / version check	`qualify` exact `runner`	Fix host or run `upgrade` after drain	Skip the canary
Queue age high but runners look idle	`queue` + `health`	Often label mismatch or repo-scoped idle capacity elsewhere — fix matching, do not add random runners	Assume “idle capacity” is fungible across repos
One bad container / leftover diagnostics	`drain` exact runner	`cleanup` → `resume`	`cleanup` without drain

Default thresholds for watchdog: queued ≥ 30 minutes, in-progress ≥ 360 minutes. Override only with a reviewed variable change.

Revive dead / sleeping / zombie runners¶

Use this when five (or any) runners are offline, missing, or “listening” but not taking work. Prefer one exact runner at a time.

flowchart TD
  A["health or report"] --> B{"Offline / missing / zombie?"}
  B -->|no| Z["Stop — no host mutation"]
  B -->|yes| C["Pick exact runner name from report"]
  C --> D["drain dry-run then apply"]
  D --> E["remediate dry-run then apply"]
  E --> F["health + drift on that runner"]
  F --> G{"Healthy?"}
  G -->|yes| H["resume dry-run then apply"]
  G -->|no| I["Escalate: adapter/SSH or stale GitHub registration ID"]
  H --> J["Next dead runner"]

Inventory the damage (read-only)
operation=health or report
scope_type=fleet, scope_value empty
dry_run=true, authorize_disruptive=false Download report.md. Note exact runner names that are offline, missing, or unexpected.
Optional: clear stuck workflow runs first If GitHub just recovered from an outage and many jobs are ancient queued / in_progress, run watchdog preview then apply (see matrix). Stale jobs can look like dead runners when the real issue is a zombie registration on the job side.
Drain one exact runner (safe even if already offline)

Field	Value
`operation`	`drain`
`scope_type`	`runner`
`scope_value`	e.g. `dnd-session-parser-docker-01`
`drain_mode`	`graceful`
`dry_run`	`true` first, then `false`
`authorize_disruptive`	`true` only on apply
`reason`	e.g. `Revive offline runner after Actions outage`

Remediate that same exact runner Same scope as drain. operation=remediate. Preview (dry_run=true) must show a non-empty plan. Apply with authorize_disruptive=true and the same reason style.
Prefer remediate for offline / missing / label drift.
Use restart only when the runner is drained, idle, and you only need a service bounce.
Applied remediate accepts one exact runner only.
Verify
health and drift with scope_type=runner and the same name.
Confirm online in the report (and in Grafana if you use it).
For homelab runners only, optional qualify canary.
Resume
operation=resume, same exact runner scope, preview then apply.
Resume removes the drain marker so jobs can match again.
Repeat for each remaining dead name. Do not widen to fleet for apply.

If remediate's plan is empty but the host is still dead, the API does not see actionable drift — escalate: Ansible/management-runner path, or a stale server-side registration that needs manual exact-ID cleanup (never bulk delete). See Failure handling.

Concise operator procedure¶

Run report for the intended scope and inspect the summary and artifact.
Run qualify on one exact runner. Expand to one placement group, then one repository, only after the canary passes.
For maintenance, select the narrowest exact scope, leave dry_run: true, keep authorize_disruptive: false, and provide the intended reason.
Review the generated plan, request fingerprint, target identities, current IDs/status, and action count. A zero-action remediate plan means no API-visible drift matched the selector.
To apply, rerun with the same exact scope, dry_run: false, authorize_disruptive: true, and a specific incident/change reason. The owner reviews the protected-environment approval.
Run health, drift, and qualify again. Confirm queue age, labels, service state, and the applied report before expanding scope.

OneUptime Stage 6 timeline notes¶

When a fleet dispatch is part of an active OneUptime incident:

Acknowledge the incident in OneUptime.
Prefer exact-runner health / drift / report with dry_run: true.
Record the GitHub Actions run on the incident via scripts/oneuptime_post_remediation_note.py or workflow oneuptime-remediation-note.yml (environment oneuptime-remediation).
Only then preview/apply disruptive operations with protected approval.

Runner fleet Prometheus alerts include action_url pointing at runner-fleet-dispatch.yml. Cross-repo qualify still fails closed from homelab (see Qualify topology limit).

Never use a fleet selector merely for convenience. The requested scope is an authorization boundary as well as a filter.

maintenance deliberately plans every selected target because GitHub's API cannot see local systemd/image drift. It remains drain-gated and serial. remediate is narrower and plans only missing, offline, label-drifted, or group-drifted targets.

Safety model¶

Dry-run is the default. An applied mutation requires all of:

dry_run: false;
the explicit boolean authorize_disruptive: true;
a reason between 10 and 500 characters;
an exact, bounded selector that resolves to configured targets;
a fresh preview whose request and plan fingerprints still match;
a fresh GitHub state recheck immediately before each action;
approval through runner-fleet-maintenance.

Applied host maintenance rejects fleet scope. Use an exact runner, repository, placement, label set, or declared organization group so a mistake cannot restart every configured worker.

Plans expire after 30 minutes by default. Changed runner ID, name, status, scope, plan fingerprint, or request fingerprint fails closed. Applied fleet mutations share one non-cancelling concurrency group, so two repair operations cannot race.

There is no runner deletion or deregistration operation. remediate accepts only one exact runner when applied, rechecks the same ID/name/status, and invokes the topology adapter to reconcile local service state. A stale server-side registration that still requires deletion is escalated for manual exact-ID recovery; never bypass this with bulk gh api DELETE.

The Linux systemd/Docker and Windows local-service adapters validate exact service/container/install paths, require drained and idle state for destructive cleanup or restart, and record idempotency results. Immediate drain requires separate immediate authorization. The Ansible adapter uses argument arrays, allowlisted executables, exact inventory host mapping, and a one-runner playbook allowlist.

For repository runners, drain first removes all custom scheduling labels from the freshly rechecked runner ID. Existing work can finish, but homelab jobs can no longer match because every workflow requires repository/class labels. The adapter then waits for Runner.Worker and stops the exact service. If adapter execution fails, the control plane attempts to restore the manifest labels. Do not introduce workflows that target only self-hosted, OS, or architecture labels; they would bypass this quarantine. Organization-runner drain fails closed until a topology adapter implements runner-group quarantine.

Watchdog¶

The watchdog treats a queued run as stale after 30 minutes and an in-progress run as stale after 360 minutes unless reviewed overrides are active. Preview records the run ID, repository, status, timestamp, age, and a precondition fingerprint.

Apply downloads that exact preview, verifies its signature and age, restricts each action to the original repository selector, refetches the run, and checks that status and timestamp are unchanged and still over threshold. Only then does it call GitHub's force-cancel endpoint. A completed, updated, or otherwise changed run is reported as failed and is not cancelled.

If GitHub is experiencing an Actions incident, leave the watchdog in preview or withhold maintenance approval. Do not repeatedly cancel and restart healthy work during a control-plane outage.

Reports and alerts¶

Every operation writes a GitHub step summary and uploads redacted artifacts for 30 days:

report.json for structured automation;
report.csv when runner rows are present;
report.md for operator review;
metrics.prom for Prometheus ingestion;
audit.jsonl for append-only event review;
plan.json for watchdog and maintenance previews.

Artifact names include the operation and workflow run identity. Secret-like keys and token patterns are redacted before reports are written. Preserve a relevant artifact outside the 30-day window when it is incident or change evidence.

Observation reports missing/offline runners, label or group drift, unexpected records, busy/online utilization, queue age, recent run/job outcomes, and API retry diagnostics. Qualification and apply failures fail their jobs, providing the normal GitHub Actions notification signal. The scheduled observation also opens or updates one runner-fleet-alert issue for failed collection, offline or unexpected runners, drift, or queue age at the configured threshold (bodies include runner names and drift issue codes, not only counts); it closes that issue after recovery. metrics.prom is portable report output; it is not automatically scraped until an operator or collector publishes it. Do not claim Alertmanager coverage from artifact generation alone.

Rollout, canaries, and rollback¶

Use this rollout order:

Observe the fleet.
Qualify one idle general runner in general-a.
Qualify one Docker runner without changing its socket boundary.
Qualify the LAN runner last.
Preview maintenance on one non-management runner.
Apply to that canary with max_parallel: 1, re-observe, then expand by placement group.

For an upgrade, drain the canary gracefully, apply only the manifest-pinned version, image tag, and release SHA-256, qualify it, and resume it. The build records version/checksum provenance labels and the adapter executes the image's Runner.Listener --version before success. Keep at least one healthy runner in each required class while rolling.

Rollback means stopping scope expansion and returning the exact canary to the last manifest-pinned configuration through Ansible. If the runner is merely drained, use resume; idempotent replay makes repeated resume safe. If a maintenance or upgrade fails, keep the target drained until host state and the report are understood. Never roll back by copying another runner's configured directory or by deleting registrations in bulk.

Ansible from CAPTAINKANGAPOO¶

Hyper-V NAT guests (172.29.240.11–.15) are reachable from the Hyper-V host, not from WSL's default network namespace and not from infra-services. Do not run ansible-playbook inside a runner guest (no ansible, no repo checkout). Do not rely on native Windows ansible-core either — it often dies at startup with OSError: [WinError 87] on stdin.

Proven control path (2026-08):

SSH key for runneradmin: ~/.ssh/github-runner-vm-admin (see ~/.ssh/gha-runner-ssh.cfg). Prefer including that snippet from ~/.ssh/config, or pass -i / -F explicitly.
Run ansible inside WSL, but tunnel TCP through Windows OpenSSH so the Hyper-V NAT route works.
Force Windows ssh.exe to ignore the user's config (-F NUL). Otherwise Include ~/.ssh/1Password/config fails non-interactively (Permission denied → terminating, 1 bad configuration options) and mid-playbook forks go UNREACHABLE.
Prefer forks = 1 and ControlMaster=no for long multi-host rolls from this path; parallel forks amplify the broken-ProxyCommand failure mode.

Example WSL SSH config fragment (~/.ssh/gha-via-windows.cfg):

Host gha-general-a 172.29.240.11
  HostName 172.29.240.11
  User runneradmin
  IdentityFile ~/.ssh/github-runner-vm-admin
  IdentitiesOnly yes
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  ProxyCommand /mnt/c/Windows/System32/OpenSSH/ssh.exe -F NUL -i C:/Users/renjamin/.ssh/github-runner-vm-admin -o IdentitiesOnly=yes -o IdentityAgent=none -o BatchMode=yes -o StrictHostKeyChecking=accept-new -W %h:%p runneradmin@%h

Use a throwaway ANSIBLE_CONFIG without ara_default if WSL ansible lacks ARA. Point ssh_args at -F ~/.ssh/gha-via-windows.cfg.

After playbook success, confirm registration in journal (Listening for Jobs) and in the target repo's Actions → Runners list. Transient Runner connect error: Conflict after --replace usually clears within a minute; persistent curl: (22) ... 404 means the host registration PAT still lacks that repository.

Failure handling¶

Symptom	Action
App token creation fails	Verify App installation, exact permission level, App ID variable, and the environment private-key secret; do not substitute the host PAT
Selector matches nothing	Compare case-sensitive runner identity and manifest repository/placement group; do not widen to fleet
Plan is stale or fingerprints differ	Create a new preview and repeat approval
Fresh-state check fails	Re-observe; another actor or GitHub changed the target
Adapter is unavailable	Verify the management runner, execution mode, Ansible inputs, SSH path, and `ansible@v1`; no host mutation occurred
Graceful drain times out	Inspect the exact job; extend a bounded timeout or obtain owner authorization for immediate drain
Qualification fails disk check	Free or expand storage; qualification requires at least 20% free space
API retries exhaust	Check GitHub status and rate limits; wait rather than bypassing controls
GitHub cancel / force-cancel returns HTTP 500	Preserve the run URL and GitHub request ID, then open a GitHub Support case; matching online runners cannot clear the orphaned run
Runner has a stale server session	Drain and remediate one exact runner after GitHub recovers; deregistration remains a manual exact-ID escalation
Registration curl 404	Add the target repo to the SOPS registration PAT allowlist; restart `github-runner@<name>`
Ansible UNREACHABLE via WSL proxy	Ensure ProxyCommand uses `ssh.exe -F NUL`; do not load the Windows user `Include`/1Password config
Allowlist assert fails	Pass only instance names that exist on the limited host (`allowlist ⊆ github_runner_instances`)

Local host health¶

Each host runs github-runner-host-health.timer every 15 minutes. It uses the SOPS-protected registration credential to check remote online state and exact labels, then checks local systemd units, runner images, stale containers, 80% disk pressure, and recent cgroup or system OOM kills. No administrative credential is stored in workflow secrets.

Inspect Linux failures with:

systemctl status github-runner-host-health.service
journalctl -u github-runner-host-health.service

A systemd restart obtains a fresh registration token and starts a clean ephemeral container. Unit shutdown stops and removes only that runner's container; startup removes an orphaned same-name container before registration. Never copy a configured runner directory between instances.