Central GitHub runner management¶

This is the authoritative runbook for observing, qualifying, and maintaining the GitHub Actions runner fleet. The central dispatcher is .github/workflows/runner-fleet-dispatch.yml; local reusable workflows perform observe (_runner-fleet-observe.yml), qualify (_runner-fleet-qualify.yml), watchdog (_runner-fleet-watchdog.yml), and maintenance (_runner-fleet-maintain.yml) operations. The scheduled workflow runs observation every six hours and the stale-run watchdog every 30 minutes.

Current recovery routing (2026-10-27): API-only dispatcher validation, observation, scheduled API checks and watchdog jobs use exact general runner homelab-ci-02 through RUNNER_FLEET_API_RUNS_ON. This separates fleet access from LAN deployment routing. The former broad LAN hold was replaced by exact-run quarantine at 10:49:47 UTC; the service is active and GitHub verifies homelab-lan-01 online. Non-deploying qualification succeeded; normal LAN job execution is restored. Watchdog apply is still an API mutation under its existing approval/environment guards; the recovery proof was read-only. See API routing recovery.

Qualification jobs still run on the selected workers. Host-maintenance preview/apply and fresh-state recheck retain the management runner (RUNNER_FLEET_MAINTENANCE_RUNS_ON, label runner-management on gha-mgmt / homelab-mgmt-01), which the Hyper-V scaler starts when those jobs queue. DEPLOY_RUNS_ON remains LAN-only. Neither selector was changed. The workflow source still falls back to DEPLOY_RUNS_ON if the API variable is absent; do not clear the recovery override without deliberate routing review, even after LAN restoration. This temporary single-instance route is not always-on redundancy. No GitHub-hosted capacity is introduced.

Zero-hosted policy: repository workflows must not use ubuntu-latest, windows-latest, or macos-latest. Enforce with scripts/check-no-hosted-runners.sh (Lint job No GitHub-hosted runners). Use CI_RUNS_ON, DOCKER_RUNS_ON, DEPLOY_RUNS_ON, RUNNER_FLEET_API_RUNS_ON, or RUNNER_FLEET_MAINTENANCE_RUNS_ON instead.

Qualify topology limit¶

Repository-scoped runners only accept jobs from their own repository. The central homelab qualify workflow therefore fails closed when the selected targets belong to another repository (for example tiktooker or dnd_session_parser). Use observe / health / drift from homelab for cross-repo visibility, and dispatch qualify from the target repository (or limit qualify scope to homelab runners) until per-repo qualify receivers exist.

flowchart LR
  Operator["Owner / protected approval"] --> Dispatch["Central dispatcher"]
  Schedule["Scheduled workflow"] --> Observe["Observe / watchdog preview"]
  Dispatch --> Observe
  Dispatch --> Qualify["Exact-runner qualification"]
  Dispatch --> Plan["Maintenance preview"]
  Observe --> API["API-only worker: homelab-ci-02 recovery routing"]
  Qualify --> Fleet["Selected runner"]
  Plan --> Approval["runner-fleet-maintenance approval"]
  Approval --> Manager["Ansible management runner"]
  Manager --> Target["Exact non-management target"]

Fleet and trust boundaries¶

The fleet manifest declares repository-scoped Linux/X64 runner instances (including homelab-docker-01 and control-plane homelab-mgmt-01). Manifest entries describe desired allocation, not proof of live registration; see the pending Gmail onboarding. General workers are ephemeral unprivileged containers. Docker workers are placed on isolated build VMs and are the only class with the Docker socket. The homelab LAN worker has host networking, a read-only operational configuration mount, and a read-write mount of the live /opt/homelab checkout. It never receives dev or dependabot labels. The management instance carries runner-management and must not carry homelab,ci (so ordinary Lint never schedules there). Applied fleet maintenance must never target homelab-mgmt-01 or host gha-mgmt.

No self-hosted runner receives the special dependabot label. GitHub-generated Dependabot update jobs require Docker access, so they remain on GitHub-hosted infrastructure. CI for Dependabot pull requests still uses the repository's ordinary ci runners. Keep Settings > Advanced Security > Dependabot on self-hosted runners disabled in every repository.

Placement¶

The four shared CI/build placement groups and dedicated canary-a placement run as on-demand Hyper-V VMs on CAPTAINKANGAPOO. The Windows host stays up; the guests Stop-VM after about 15 minutes idle and start when a matching job queues. First job after sleep typically waits 1–3 minutes for boot, Docker, and the listener. The LAN runner remains always-on on infra-services (placement:lan-a, inventory ansible_user: someone; the fleet apply key must be authorized there so homelab-mgmt-01 can reach it when that guest is awake). The management control-plane VM is gha-mgmt (see Create gha-mgmt Hyper-V guest); it sleeps with the rest of the fleet and wakes for qualify / maintain apply.

Current Hyper-V dynamic-memory targets (after the 2026-08-05 capacity grow) still apply while a guest is running:

VM	Startup	Min	Max
`gha-general-a` / `gha-general-b`	12 GiB	6 GiB	28 GiB
`gha-docker-a` / `gha-docker-b`	10 GiB	4 GiB	16 GiB
`gha-canary-a`	4 GiB	2 GiB	4 GiB
`gha-mgmt`	4 GiB	2 GiB	8 GiB

Raising startup/minimum requires a brief VM stop; maximum can change while running. Host-specific github_runner_instances in infra/ansible/inventory/github-runners.yml contain only the entries assigned to each host. Placement → VM names live in infra/github-runners/hyperv-scaler/placements.yml.

Homelab Docker CI (DOCKER_RUNS_ON) uses instance homelab-docker-01 on gha-docker-b — an additional ephemeral docker-class worker, not a new Hyper-V guest.

Telegram Docker CI uses telegram-docker-01 on gha-docker-b. Its repository variable must be DOCKER_RUNS_ON=["self-hosted","Linux","X64","Telegram","docker"]; keep ordinary lint and test jobs on CI_RUNS_ON so only service-container work crosses the Docker socket boundary.

Gmail runner onboarding (proposed)¶

This is a draft provisioning proposal, not proof of deployed capacity. 2026-10-27 current limitation: Gmail's DOCKER_RUNS_ON variable is absent and its proposed Docker runner is unregistered. This is a separate provisioning issue, not fallout from the edge quarantine; observed Gmail Python/Maintenance successes do not qualify the Docker lane. Add only two repository-scoped Linux/X64 instances for spadoople/gmail-order-tracker-bot:

Instance	Class	Existing VM / placement	Inherited container memory limit
`gmail-order-tracker-ci-01`	general	`gha-general-a` / `general-a`	4 GiB
`gmail-order-tracker-docker-01`	docker	`gha-docker-a` / `docker-a`	6 GiB

Before this addition, the checked-in inventory allocates 12 instances to each shared general VM and four Docker instances to gha-docker-a versus five to gha-docker-b. Choosing general-a follows the usual first-general-instance placement; choosing docker-a balances the declared Docker instance counts. These counts and per-container limits do not establish spare CPU, memory, disk, or concurrent-build capacity. The owner must review real capacity and shared-build-VM isolation before merge/apply. No VM, resource limit, existing runner, registration credential, or production workload is changed by this proposal. One runner per class is the minimum, with no same-class redundancy.

After registration and qualification, set these exact repository variables (the values below are JSON arrays; store each array as that variable's value):

{
  "CI_RUNS_ON": ["self-hosted", "Linux", "X64", "gmail-order-tracker-bot", "ci", "gmail-order-tracker-ci-01"],
  "DOCKER_RUNS_ON": ["self-hosted", "Linux", "X64", "gmail-order-tracker-bot", "docker", "gmail-order-tracker-docker-01"]
}

The exact instance labels keep ordinary CI off Docker: the docker-class runner also has ci, so a selector ending at gmail-order-tracker-bot,ci is too broad. Never substitute telegram_companion, homelab, LAN, management, or bare self-hosted/Linux/X64 selectors. General jobs get no Docker socket; Docker jobs use the existing isolated build VM, never the production Docker daemon. Neither class receives Gmail/OAuth state, deployment keys, or production mounts.

Owner gates, in order (none completed by publishing this proposal):

Approve capacity and PR trust/isolation on the two existing VMs, review CI, and separately authorize merge. A draft PR is not permission to apply.
Through the existing protected administration process, include Gmail in the existing host registration credential's repository allowlist and verify selected-repository access for Fleet Read, Operator, and Watchdog Apps. Confirm the scaler/observability App can read Gmail too. Do not extract, replace, or copy credentials into Gmail or workflow secrets.
After separately authorized rollout, use the existing runner playbook with one host and one exact github_runner_instance_allowlist per invocation: gha-general-a / gmail-order-tracker-ci-01, then gha-docker-a / gmail-order-tracker-docker-01. Review shared role/host effects; do not apply to the whole fleet or restart unrelated instances.
Refresh the scaler's pinned runtime checkout/map through its owner-operated procedure, and roll out the collector allowlist in services/monitoring/compose.yml. Merely merging generated artifacts does not refresh the scaler runtime or running collector; the expected fleet can report missing runners until registration succeeds.
Verify actual GitHub labels/scope, wake-on-queue, Python 3.12, Docker/Buildx, and synthetic no-network application smoke checks. Qualify from Gmail: central homelab qualification fails closed for other repository-scoped runners. Only then set the variables above and verify Gmail's PR checks. Do not dispatch production deployment, publish images, or enable the special dependabot runner label. Dependabot update jobs remain separate from PR CI.

Dependency Steward enrollment is tracked separately in Steward PR #37. Its App repository access and merge-mode activation are not established or changed by this runner proposal. The Gmail maintenance PR link belongs in this provisioning PR once the parent work publishes it.

On-demand Hyper-V scaler¶

GitHub only assigns work to online runners. A hidden scheduled task on the Hyper-V host polls queued/in-progress jobs (~15s), starts the placement VMs that can take them, and ACPI-stops guests after idle_timeout_seconds (900) with no needed or busy runners. Queued jobs older than max_queued_age_seconds (86400) are ignored so GitHub ghosts cannot pin guests awake.

Stays up	Sleeps
Windows workstation, scaler task, `homelab-lan-01`	`gha-general-a/b`, `gha-docker-a/b`, `gha-canary-a`, `gha-mgmt`

Do not run Invoke-HyperVRunnerScaler.ps1 in an interactive terminal. The installer registers a background task (-WindowStyle Hidden, S4U principal when the account may log on as a batch job) and pins a runtime checkout at C:\ProgramData\homelab\hyperv-scaler\src so a Documents feature-branch checkout cannot break wake/sleep.

Do not Stop-VM the fleet until the scaler has woken a guest for a real queued workflow. Otherwise jobs sit in queue until GitHub times them out.

Owner install on CAPTAINKANGAPOO (elevated PowerShell 7, Hyper-V Administrators, uv and git on PATH):

Copy a read-only GitHub App PEM to C:\ProgramData\homelab\hyperv-scaler\fleet-read.pem (icacls for your user only). Do not use gh auth token as the reboot path: that token is not in Windows Credential Manager after a reboot, and the scaler process only keeps working until the next boot because it caches GH_TOKEN in memory.
Edit C:\ProgramData\homelab\hyperv-scaler\config.ps1 with HYPERV_SCALER_GITHUB_APP_ID, HYPERV_SCALER_GITHUB_APP_PRIVATE_KEY_FILE, and HYPERV_SCALER_GITHUB_APP_INSTALLATION_ID (or a GH_TOKEN with Actions read on every fleet repository).
From an elevated pwsh in the repo (any branch is fine; the installer clones HEAD into ProgramData):

.\infra\github-runners\hyperv-scaler\Install-HyperVRunnerScaler.ps1

That syncs the runtime checkout, stops any stray invoke processes, registers HomelabHyperVRunnerScaler (boot + logon + 5-minute keep-alive, IgnoreNew, no 72-hour limit), and sets AutomaticStartAction=Nothing on the six guests. It does not power them off. 4. Dry-run from the runtime tree: $env:HYPERV_SCALER_DRY_RUN='1' then C:\ProgramData\homelab\hyperv-scaler\src\infra\github-runners\hyperv-scaler\Invoke-HyperVRunnerScaler.ps1 -Once. 5. Prove a wake with one stopped guest and a matching queued job, then allow idle stop for the rest of the fleet.

Live (2026-10-05)¶

Task HomelabHyperVRunnerScaler runs as CAPTAINKANGAPOO\renjamin against C:\ProgramData\homelab\hyperv-scaler\src, not the Documents working tree. Auth is the GitHub App PEM at C:\ProgramData\homelab\hyperv-scaler\fleet-read.pem, with the App ID and installation ID in config.ps1. That PEM is a copy of the read-only github-actions-collector key (services/monitoring/secrets/github-app.pem on infra-services). After the 2026-10-05 reboot, gh auth token returned no oauth token, config.ps1 threw before the log loop, and the task exited 1 every five minutes while scaler.log stayed frozen. The previous process had looked healthy only because it still held a token in memory.

Installation access tokens last one hour. The scaler mints a new one every 45 minutes. It must not leave the first token in GH_TOKEN for the life of the process, or the next hour is a stream of 401s until reboot.

config.ps1 is loaded inside the poll loop. A config or token failure is appended to scaler.log and retried on the next tick. The long-running loop can still exit 0 and leave the task Ready; a 5-minute keep-alive trigger starts it again when no instance is running. Ready for more than one interval with fresh queued fleet work is still a scaler failure.

Each tick failure is appended to C:\ProgramData\homelab\hyperv-scaler\scaler.log including Python stderr from python.stderr. Corrupt state.json is treated as empty idle timestamps. A successful tick writes last-success.json. One guest failing to ACPI-stop no longer aborts the rest of the tick; the scaler then Stop-VM -TurnOff.

The installed task launches PowerShell with -WindowStyle Hidden. Closing a Windows Terminal window must not be part of operating the fleet.

S4U lets the task start at boot without an interactive token. If Windows rejects S4U (missing “Log on as a batch job”), the installer falls back to Interactive and the logon trigger still starts the loop after sign-in. Verify triggers after installation:

(Get-ScheduledTask -TaskName HomelabHyperVRunnerScaler).Triggers |
    Select-Object CimClass

The output must contain MSFT_TaskBootTrigger, MSFT_TaskLogonTrigger, and MSFT_TaskTimeTrigger (the 5-minute keep-alive).

Wake proof: gha-mgmt was stopped, a qualify dispatch queued runner-management, the scaler started the guest (~1 minute Off→Running), and run 32547732271 succeeded. At that historical checkpoint, RUNNER_FLEET_API_RUNS_ON matched DEPLOY_RUNS_ON; the 2026-10-27 recovery override below supersedes that routing.

Historical GitHub run aiProject 32211882200 is a queued ghost (created 2026-08-19; cancel returns 409 “not in progress”, delete returns 403). aiproject-ci-01 was listening and GitHub still would not assign it. The scaler ignores queued jobs older than 24h so that ghost cannot keep gha-general-b awake. This August 19 ghost predates and was not caused by the October 27 recovery work. GitHub does not expose a supported API to delete a queued-but-not-in-progress run.

Ansible to 172.29.240.11–.16 still only works from CAPTAINKANGAPOO / gha-mgmt (Ansible from CAPTAINKANGAPOO). Keep maintain apply on RUNNER_FLEET_MAINTENANCE_RUNS_ON.

Idle Hyper-V runners looking offline in GitHub is expected. Prometheus GitHubRunnerOffline / GitHubRunnerServiceStopped / Hyper-V NodeExporterUnavailable / RunnerTelemetryUnavailable ignore on-demand placements (general-a/b, docker-a/b, canary-a, mgmt-a). GitHubActionsQueueLong and GitHubRunnerNoIdleCapacity (10 minutes) remain the scaler-failure signal.

The canonical control-plane manifest is infra/github-runners/fleet.yml. It defines expected identity, repository, labels, placement group, pinned runner version, adapter contract, thresholds, and explicit deregistration prohibition. Ansible inventory continues to define the physical host mapping. A runner name must map to exactly one inventory host before maintenance can run.

Current topology assumptions:

Every configured instance is repository-scoped, linux-x64, and uses the ansible@v1 maintenance adapter.
Repository-scoped runners cannot belong to GitHub organization runner groups. Use placement scope for this fleet's general-a, docker-b, and similar topology groups. group is reserved for an explicitly declared organization runner group.
Organization-scoped runners and groups are supported by the API model only when explicitly represented in a future manifest. They require organization permissions and must not be inferred from repository runners.
The Linux reference host adapter manages systemd units and Docker containers. A Windows local-service adapter defines the same bounded v1 contract, but no Windows runner is currently present in the canonical manifest. Windows reporting and qualification are ready; applied Windows maintenance requires a reviewed transport adapter that invokes the local PowerShell contract with exact service/install identities. The built-in ansible@v1 registry rejects that topology. macOS and ARM are selection vocabulary, not deployed topology.

Registration credential¶

Create infra/ansible/inventory/group_vars/github_runner_hosts/registration.sops.yaml from the adjacent example and encrypt it with SOPS before committing or deploying. The value is never a workflow secret and must not appear in normal inventory, logs, or shell history. Use a fine-grained PAT restricted to runner administration on every repository that appears in infra/github-runners/fleet.yml. When adding a new repo's instances, extend that PAT's repository allowlist before expecting registration to succeed. A missing repo usually returns HTTP 404 (not 403) from /repos/{owner}/{repo}/actions/runners/registration-token, and the unit crash-loops with curl: (22) The requested URL returned error: 404.

This host registration PAT remains inside the SOPS/host boundary. Do not copy it into GitHub Actions. Central management uses a dedicated GitHub App with short-lived installation tokens; it does not replace the host credential used to obtain runner registration tokens.

Configure the control plane¶

Dedicated GitHub Apps¶

Use separate Apps so a read-only or unattended workflow never receives a private key capable of broader mutation:

Runner Fleet Read — Actions read, repository Administration read, organization Self-hosted runners read, and Metadata read.
Runner Fleet Operator — repository Administration write, organization Self-hosted runners write, and Metadata read. This key exists only in the reviewer-protected maintenance environment. Write access removes scheduling labels before drain; there is no runner-delete operation.
Runner Fleet Watchdog — Actions write and Metadata read only. This key can force-cancel stale runs but cannot administer runner registrations.

Install each App only on managed repositories. Organization Self-hosted runners permission is needed only when organization-scoped runners/groups enter the manifest. Repository-scoped runners cannot become members of those groups.

Configure:

Keep Issues enabled in the control repository so scheduled alerts can reconcile the runner-fleet-alert issue with GITHUB_TOKEN.
Repository variables RUNNER_FLEET_READ_CLIENT_ID, RUNNER_FLEET_OPERATOR_CLIENT_ID, and RUNNER_FLEET_WATCHDOG_CLIENT_ID (GitHub App Client ID, Iv1.…). Fleet workflows pass these as client-id to actions/create-github-app-token. Do not leave them empty or token mint fails. Numeric RUNNER_FLEET_*_APP_ID vars may still exist for older tooling (Hyper-V scaler uses App ID locally); they are not used by fleet workflows. The spadoople-runner-observability App is SOPS on infra-services, not a repo variable.
RUNNER_FLEET_READ_APP_PRIVATE_KEY only in runner-fleet-readonly.
RUNNER_FLEET_OPERATOR_APP_PRIVATE_KEY only in runner-fleet-maintenance.
RUNNER_FLEET_WATCHDOG_APP_PRIVATE_KEY in runner-fleet-maintenance and runner-fleet-automation.
Repository variable RUNNER_FLEET_API_RUNS_ON as a JSON runner-label array. Current recovery value: ["self-hosted","Linux","X64","homelab","ci","homelab-ci-02"]. The exact instance label prevents API jobs from landing on Docker or LAN runners. Observe, watchdog (including guarded apply), dispatcher validation and scheduled API checks use this; source still falls back to DEPLOY_RUNS_ON if unset.
Repository variable RUNNER_FLEET_MAINTENANCE_RUNS_ON as a JSON runner-label array: ["self-hosted","Linux","X64","runner-management"]. Required for qualify and maintain apply — there is no ubuntu-latest fallback. Guest gha-mgmt and instance homelab-mgmt-01 are live (2026-08-09, 172.29.240.15) and on-demand after the scaler is installed.
Repository variable RUNNER_FLEET_ANSIBLE_PRIVATE_KEY_FILE — absolute path inside the homelab-mgmt-01 container to an SSH private key that can log in as runneradmin on the other Hyper-V runner guests (same key as ~/.ssh/github-runner-vm-admin on CAPTAINKANGAPOO). Live path: /etc/homelab/runner-fleet/id_ed25519 (host file, ACL’d for runner UID 10001, mounted by launch-runner for management class). Do not point this at /home/runneradmin/.ssh/... — fleet jobs run as container uid 10001 and cannot see the host home directory (no such identity / Permission denied (publickey)).
Fleet apply sets ANSIBLE_CONFIG=infra/ansible/ansible-fleet.cfg (host-key checking off, roles_path, no ara). The adapter cwd is the repo root, so infra/ansible/ansible.cfg is not auto-loaded; without the fleet cfg, apply fails with Host key verification failed even when the key works.
Optional repository variable for adapter timeout overrides as documented elsewhere.

Keep all private keys in environment secrets rather than repository-level secrets. Rotate one App at a time by adding its new key, updating only its listed environments, verifying the affected operation, and deleting the old key.

Protected environments¶

Create these GitHub environments:

runner-fleet-readonly for API discovery, matrix construction, and plan generation. Restrict deployment branches to protected main; do not expose the read App key to feature-branch workflow code.
runner-fleet-maintenance for operator-requested host mutation. Require the owner as reviewer, prevent self-review where available, and restrict deployment branches to the protected default branch.
runner-fleet-automation for the existing scheduled stale-run watchdog. Do not add a required reviewer, because that would leave a deployment waiting every 30 minutes. Restrict it to the default branch and store the Actions-only watchdog key.

Environment approval is a required apply gate, not a substitute for the request's authorization fields. Owner approval is required for live disruptive maintenance. Scheduled cancellation is explicit policy in the scheduled workflow and still uses preview, age thresholds, fingerprints, and a fresh per-run recheck.

The dispatcher and scheduler use secrets: inherit when calling the local reusable workflows. GitHub otherwise resolves secrets from the reusable job's environment as empty. Keep these callers and reusable workflows in this repository on protected main; inheritance also makes caller-visible repository and organization secrets available to the called workflow.

Management runner prerequisites¶

The runner selected by RUNNER_FLEET_MAINTENANCE_RUNS_ON must be a trusted, dedicated management path with:

a clean checkout of the default branch and access to Python package installation; the workflow pins Python, PyYAML, and ansible-core;
both generated and GitHub-runner Ansible inventories;
the maintenance playbook and installed ansible@v1 adapter;
SSH and sudo reachability to the exact target host, without exposing its key to ordinary CI runners;
RUNNER_FLEET_ADAPTER_MODE=execute and RUNNER_FLEET_CONTROL_RUNNER_NAME set to its exact runner name;
the Hyper-V scaler able to start gha-mgmt when a runner-management job queues (the guest is not 24/7 once on-demand is live).

Homelab maintenance must never target the management runner executing the operation. The adapter rejects self-maintenance, and operators must choose a repository, placement-group, or exact-runner scope that excludes it. Do not run fleet-wide applied maintenance until a second management runner provides a non-self target path.

Create gha-canary-a Hyper-V guest¶

Status (2026-08-29): Planned owner task on CAPTAINKANGAPOO. The committed target is VM gha-canary-a, address 172.29.240.16, placement canary-a, and runner homelab-ephemeral-canary-01.

This VM exists only to prove the OneUptime start_ephemeral_runner operation. It must contain exactly one repository-scoped, non-management runner with labels runner-remediation-canary, dev, and homelab-ephemeral-canary-01. Never add generic homelab, ci, deploy, docker, or runner-management labels.

Confirm 172.29.240.16 is unused from CAPTAINKANGAPOO.
Clone a general-class sibling such as gha-general-b on the same vSwitch. Do not clone gha-mgmt, because its control-plane key mounts and secrets must never enter the canary.
Rename the guest to gha-canary-a; set dynamic memory to 4 GiB startup, 2 GiB minimum, and 4 GiB maximum.
Before connecting it to the runner network, regenerate machine ID and SSH host keys, remove every cloned github-runner@* unit/container, and set the static address to 172.29.240.16 using the sibling gateway and DNS values.
Run the runner-host Ansible play against gha-canary-a. The committed inventory installs only homelab-ephemeral-canary-01.
Verify the runner registers with only its exact labels, stop the VM, and confirm that an exact-label queued job wakes only gha-canary-a.

Do not enable the remediation workflow until the wake proof, offline-state health evidence, and post-start effectiveness check all pass.

Create gha-mgmt Hyper-V guest¶

Status (2026-08-09): Guest exists and is online — gha-mgmt at 172.29.240.15, runner homelab-mgmt-01 listening with runner-management. Steps below are historical recreate notes only.

Repo automation installs runners on existing Linux guests only. Creating gha-mgmt is an owner Hyper-V task on CAPTAINKANGAPOO (Administrator PowerShell). Prefer cloning a known-good sibling rather than a bare ISO.

0. Inspect siblings (do this first)¶

Get-VM -Name gha-docker-b | Format-List Name, State, Generation, Path, Status
Get-VMNetworkAdapter -VMName gha-docker-b | Format-List *
Get-VMMemory -VMName gha-docker-b | Format-List *
Get-VMHardDiskDrive -VMName gha-docker-b | Format-List *
Get-VMSwitch | Format-Table Name, SwitchType

Note the vSwitch name, Generation, and VHDX path. Inside the guest (SSH as runneradmin@172.29.240.14) also record:

ip -br a
ip route
cat /etc/resolv.conf
hostnamectl

Use the same prefix/gateway/DNS for gha-mgmt. Planned address: 172.29.240.15 (confirm unused).

1. Preferred: clone gha-docker-b¶

Shut down or live-export gha-docker-b only if required by your Hyper-V version; otherwise use Export-VM / Import-VM / Copy-VM as available on your host.
Create VM name gha-mgmt from that export/copy on the same vSwitch.
Set dynamic memory: Startup 4 GiB, Minimum 2 GiB, Maximum 8 GiB.
First boot: change hostname to gha-mgmt, set static IP .15, regenerate machine-id and SSH host keys (sudo rm -f /etc/ssh/ssh_host_* then sudo dpkg-reconfigure openssh-server or equivalent), clear any old github-runner@* units that still point at cloned instance names.
Confirm SSH: ssh runneradmin@172.29.240.15.
Ensure /etc/homelab/age-key.txt exists if siblings use SOPS on-host (copy from a sibling only via your normal secret path — do not commit it).

2. Alternate: Ubuntu 24.04 Gen2 from ISO¶

Only if you have no clone path: Gen2 VM, same vSwitch, Ubuntu Server 24.04, user runneradmin with SSH key auth + passwordless sudo, install Docker CE matching siblings, place age key, static IP .15.

3. Register via Ansible (after inventory is on main / this branch)¶

Inventory host gha-mgmt and instance homelab-mgmt-01 are already declared in infra/ansible/inventory/github-runners.yml and infra/github-runners/fleet.yml. Labels must include runner-management and must not include homelab or ci.

cd /opt/homelab
ANSIBLE_CONFIG=infra/ansible/ansible.cfg ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml \
  --limit gha-mgmt

4. GitHub variable¶

Repo Settings → Secrets and variables → Actions → Variables:

RUNNER_FLEET_MAINTENANCE_RUNS_ON = ["self-hosted","Linux","X64","runner-management"]

Confirm homelab-mgmt-01 is online with those labels, then re-run any queued fleet workflows.

Thresholds and overrides¶

The manifest defaults are:

Setting	Default	Actions variable / environment override
In-progress run stale age	360 minutes	`RUNNER_FLEET_STALE_RUN_MINUTES`
Queued run stale age	30 minutes	`RUNNER_FLEET_STALE_QUEUE_MINUTES`
Observation activity window	24 hours	`RUNNER_FLEET_RECENT_RUN_HOURS`
Maximum selected targets	32	`RUNNER_FLEET_TARGET_CAP`
GitHub API retry attempts	4	`RUNNER_FLEET_RETRY_ATTEMPTS`

RUNNER_FLEET_PLAN_TTL_MINUTES defaults to 30, and request parallelism is bounded by RUNNER_FLEET_MAX_PARALLEL (default 8). Prefer changing the canonical manifest through review. RUNNER_FLEET_ADAPTER_TIMEOUT_SECONDS defaults to 1800 for host work. Use overrides only for a time-bounded incident, record the reason, and restore defaults afterward.

Deploy¶

Build the isolated general and Docker VMs and confirm the LAN host is reachable.
Review infra/ansible/inventory/github-runners.yml and confirm its host addresses and instance allocation match the deployed fleet.
Create and SOPS-encrypt infra/ansible/inventory/group_vars/github_runner_hosts/registration.sops.yaml from the adjacent example. Commit only the ciphertext.
Run the playbook with both inventory sources (see Ansible from CAPTAINKANGAPOO when operating from the Hyper-V host):

cd /opt/homelab
ANSIBLE_CONFIG=infra/ansible/ansible.cfg ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml

ANSIBLE_CONFIG is what sets roles_path. From a repo-root checkout it is not loaded automatically. On CAPTAINKANGAPOO, use the WSL path in Ansible from CAPTAINKANGAPOO and a throwaway config if that Ansible has no ARA callback.

For a single new instance, limit to its host and pass an allowlist that contains only names present on that host (the role asserts allowlist ⊆ host instances):

cd /opt/homelab
ANSIBLE_CONFIG=infra/ansible/ansible.cfg ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml \
  --limit gha-docker-b \
  -e '{"github_runner_instance_allowlist":["rexit-evolved-docker-01"]}'

Set repository variables:
CI_RUNS_ON=["self-hosted","Linux","X64","<repo>","ci"]
DOCKER_RUNS_ON=["self-hosted","Linux","X64","<repo>","docker"]
DEPLOY_RUNS_ON=["self-hosted","Linux","X64","homelab","lan","deploy"]
Dispatch runner qualification before moving CI workloads.

The LAN container is configured to run as lanrunner UID/GID 1001, matching the owner of the live checkout and private Komodo API file. Its launcher mounts /opt/homelab read-write and retains /etc/homelab:ro for labctl, including the existing read ACL on /etc/homelab/labctl.env. That file ACL does not solve parent traversal. General and Docker runners continue to run as UID/GID 10001 and never receive the live checkout mount. No Docker socket or Docker group membership is added to LAN.

Historical failure (2026-10-26 UTC): deploy run 40299611768 failed loading /etc/homelab/komodo-deploy.env before host fast-forward, Sync, Procedure execution or new pending intent. The file was 1001:1001, 0600, but active Ansible convergence alternated its parent between root:docker 0750 (unreadable to the runner) and root:root 0755 (readable). The actual container used 1001:1001 plus supplementary GID 10002, not host Docker GID 987. Earlier point-in-time authentication is not proof of stable access.

PR #508 merged; credential migration complete: the existing pair was migrated unchanged to /etc/komodo-deploy/komodo-deploy.env, with verified 1001:1001, 0600 metadata under a dedicated root:1001, 0710 parent. The legacy file, ciphertext and age identity were retained. Group 1001 has search/traversal only, no directory listing or writes. The renderer rejects wrong parent owner/group/mode, extended/default ACLs and ancestor symlinks even in check/unchanged paths. Unsafe existing directories are refused rather than repaired. The shared /etc/homelab cross-role modes and labctl traversal remain a separate unresolved issue; do not chmod that parent, change those roles or add the runner to Docker.

With github_runner_komodo_deploy_api_enabled: true, both preflight and actual LAN runner receive --mount type=bind,src=/etc/komodo-deploy,dst=/etc/komodo-deploy,readonly. This fails on a missing source. Keep a directory bind, not a file bind or auto-creating volume shortcut: atomic replacement must be visible without pinning the old inode. The runner-role gate checks SOPS/output consistency before runner configuration changes; container preflight validates metadata and readability without SOPS or an age key.

The prior approved cutover checkpoint records host /opt/homelab at 9240794 (#506/#508 merged, #509 retained), no relay pending intent, deployment workflow disabled, and homelab-lan-01 offline with busy=false, no container and its drain marker present. Temporary runner-identity read-only preflight passed; actual recreated-runner checks had not yet occurred at that checkpoint. Ansible timers were active and the owned maintenance marker was removed; the dated apply/check results do not establish all-convergence-checks-passed. No ResourceSync, deployment or Core restart was performed.

Historical runner-qualification phase: runner_qualified_bootstrap_pending (2026-10-26 19:05 UTC). The owner merged PR #510 at 2026-10-26 11:27:39 UTC as 0416d8e (reviewed activation/host revision); all 26 PR-head checks succeeded. Review, activation and scoped credential/both-mode/natural-convergence acceptance are complete. See natural-convergence acceptance complete. The earlier 180-second wait (exit 75: still busy) is historical.

Historical activation on 2026-10-26 UTC, through the 12:06 capture (not instructions to replay or evidence of the later acceptance):

The owner allowed up to 30 minutes of waiting. The existing normal apply, started at 11:22 UTC, finished 12:01:20 UTC, exit 0, without interruption. The check service's last 11:49 UTC exit 0 may be a lock skip and does not prove full convergence. A new window paused both timers with an owned maintenance marker. Native read-only checks found 10 Procedures, 6 Repos, 21 Stacks, 0 Actions and the Sync idle, with no nonterminal Updates.
Under pull + relay locks, /opt/homelab fast-forwarded 9240794 → 0416d8e, preserving all four tracked local edits and all observed host-local paths, without staging/reset. Routine ansible-pull uses a separate checkout; this advance was verified independently. The three persistent flags common_komodo_deploy_api_reviewed, common_render_komodo_deploy_env and github_runner_komodo_deploy_api_enabled remain true only in infra-services host_vars; role defaults remain false, and one-off extra vars are insufficient.
The reviewed one-off wrapper applied common with tasks_from: komodo-secrets and tag komodo-deploy-credentials: ok=8 changed=0 skipped=2 failed=0; renderer check and validate both exited 0. Then ONLY github_runner, with github_runner_instance_allowlist: [homelab-lan-01] and no github_runner_observability: ok=39 changed=3 skipped=6 failed=0. Changes were the installed launcher, runner GC helper file and service start—not an image build, GC operation or Core change. Earlier tag/task listing and five activation tests passed. Main site.yml excludes github_runner; normal convergence alone does not install its enabled launcher. The credential pair was not recaptured or rotated; renderer status remains 0 unchanged/valid, 3 changed/would change, 1 failure, and --check never mutates files.
The owned runner drain marker stayed through apply and was released at 12:05:55 UTC after a fresh queue check. The waiting launcher continued without a redundant restart. The 90-second GitHub registration wait expired; immediate follow-up found homelab-lan-01 online/idle, runner ID 24873, with zero service restarts, no launcher errors and no registration retry.
The actual container (ID prefix f469fae) started 12:06:26 UTC, image 2.336.0-r5, default UID:GID 1001:1001, groups [1001, 10002], no GID 987 or Docker socket, cap-drop=ALL and no-new-privileges. Its same-path /etc/komodo-deploy directory mount is read-only. Host SOPS check and actual container validation exited 0; default-identity read succeeded, directory listing was denied, and write-open returned EROFS. GET /user authenticated enabled homelab-deploy, non-admin/non-superadmin. No identity override was used for these actual-runner checks.

At that 12:06 historical checkpoint, /etc/homelab had been observed only as root:root 0755; both-state/natural-convergence acceptance remained open. The zero-restart runner 24873/container f469fae observation is not current identity.

By 18:24 UTC, eight normal applies since activation had successful full recaps, latest 18:12:11–18:23:15 UTC, ok=439 changed=29 skipped=98 failed=0. At that capture, runner 25235, container cdd2e124a9d2, had started 15:09:30 UTC; four service restarts since activation followed clean listener exit 0 and re-registration, not failed credential preflight.

Scoped acceptance passed — 19:05 UTC: the read-only observer (18:26:15–19:05:00 UTC, within its one-hour bound) watched the next natural apply 18:54:43–19:04:57 UTC, exit 0, full recap ok=439 changed=29 skipped=98 failed=0. The same container's default UID:GID 1001:1001, groups [1001, 10002], without Docker GID 987/socket, retained cap-drop=ALL, no-new-privileges and the same-path read-only directory mount. Probes passed before 18:26:17 (root:root 0755), at 18:57:38 (root:docker 0750, natural transition 18:57:37) and after at 19:05:00 (root:root 0755, returned 19:01:44), with the shared parent stable before/after each probe. Each host SOPS check and actual-container validation exited 0; credential read succeeded, listing was denied, write-open returned EROFS, and GET /user identified enabled, non-admin/non-superadmin homelab-deploy. No secrets were output. No chmod, forced apply, timer pause, maintenance/drain marker, restart, rotation, API execution or resource/grant change was used to obtain acceptance. Detailed evidence: canonical acceptance record.

This is not whole-host check-mode acceptance: the separate 17:50–17:51:58 UTC run exited 2 at operator-shell : Set operator shell home (undefined value; ok=135 changed=3 failed=1 skipped=76). That failure remains unfixed. At the 2026-10-26 19:05 UTC capture, both timers were active, with no maintenance/drain markers or relay pending intent; the deployment workflow was disabled. Core/Periphery retained their original 2026-10-25T11:33 start times. External Sync (10 Procedures / 0 Actions → 26 / 2), grants (36 Read + 2 Execute → 51 Read + 5 Execute), optional image-publisher App/Core PAT and workload deployment were then separate approval gates; none was enabled in that acceptance window. Preserve unrelated host files and /opt/homelab/.git/komodo-deploy/ lock/pending recovery state.

Historical bootstrap guard — 2026-10-26 20:24 UTC: PR #513 merged at 20:20:07 UTC as 645bc3c5467f82ef749048ef296b457a48476e39, with all 26 checks passed. Native pending cache naturally refreshed to 645bc3c at 20:24 UTC, remote-content SHA-256 2551c36fe2b846ab5123e0c52f1a6823cc36f2c6ac3bbeaa2b84c0ca8e8b1c50: 18 creates + 13 updates, no deletes/deploy targets/errors/Variable/UserGroup changes, all four new retry flags false in merged source and PENDING preview, not live-created Procedures. At that capture, phase was runner_qualified_bootstrap_pending, host was 8bf7aa5, normal Ansible apply was left alone, and five-profile preparation approval was pending. Source-only Inputs.snapshot(rendered=False) passed all six profiles without SOPS decryption/output writes; LiteLLM/Reddit checks both exited 0, not DB-auth proof. The unsafe c9a0fd7 19:25 preview is also historical. Sync was then unexecuted; its enabled webhook means workflow disablement is not a global lock. Six legacy scoped webhooks and 36 Read + 2 Execute grants were unchanged at that capture.

Current checkpoint — 2026-10-27: edge_dispatch_blocked_github_queue. The owner-approved deploy-reviewed-images-edge dispatch after #517 merged was accepted at 07:47:46 UTC: run 40469274207, attempt 1, head 35f9a67d218f78251cedb2dd26a1e1c9edfceee1. Brief enablement followed by immediate disablement preceded a bounded 20.5-minute watch ending queued, zero jobs/check runs. A 60-second re-enable did not advance it. Cancel/force-cancel both returned HTTP 409, Cannot cancel a workflow run that has not been queued yet.; in-place rerun returned HTTP 403, This workflow is already running. No retry was accepted or new dispatch made. Immediate disablement may have contributed; cause is unconfirmed, not evidence of server failure or runner misconfiguration. Edge execution is not accepted or a no-op.

Historical broad containment — 08:14 UTC, superseded at 10:49:47 UTC: workflow was disabled_manually; owned drain marker /var/lib/github-runner/drain/homelab-lan-01 recorded 40469274207. The idle LAN runner was stopped only after GitHub busy=false and no Runner.Worker were verified: service failed/stopped, container absent, no job interrupted. Both Ansible pull timers were restored active and the owned /etc/homelab/MAINTENANCE removed. At that capture, host HEAD was unchanged at 0f522f0, with no relay pending intent or edge RunProcedure (only old CreateProcedureUpdate); Sync remained bootstrap 824c148. Edge app/Core/Periphery container IDs/start times were unchanged. Prior runner qualification, bootstrap and grants remain complete: 51 Read + 5 Execute, 26 Procedures + 2 Actions.

Current operator boundary: the service is active under the exact-run guard, not drained/offline. GitHub verifies it online; qualification run 40491873440 succeeded and normal LAN jobs are restored. The original run is still queued with zero jobs; keep komodo-deploy.yml disabled. No redispatch, cancel/rerun retry or direct UI Procedure/Stack bypass. Browser access had returned private-page 404 despite authenticated gh API access; reconcile through GitHub inspection/support without repeated mutation attempts. Guard removal needs proof the orphan is terminal with no executable pending work and a deliberate reviewed follow-up—not merely runner restoration. The owner merged docs-only recovery PR #519 as 4b2cb0c; rollout holds remain. See edge dispatch blocked before job creation.

Historical bootstrap checkpoint — 2026-10-27 07:16:31 UTC: bootstrapped_deployment_pending. The separately authorized configuration bootstrap is complete: same-key Reddit fingerprint-only refresh (no env render/key change), then all seven input checks passed. One external RunSync using existing bot scoped Execute completed exact Update 6ae04f5aa10ff54a5801dc65 successfully at 07:14:52 UTC, applying reviewed main 824c1487735a5ea0b8d25a1aafab8f0362d6fa92. Resource bytes were unchanged from #513; Sync last_hash=824c148, zero pending resource changes/deploy targets, no errors. Existing admin browser/native UI API added exactly 15 Read + 3 Execute, preserving the original 38 grants. Root bot readback verified 51 Read + 5 Execute = 56, unchanged non-admin identity and empty user-specific/base permissions; no admin or Server/Build creation rights added. Live sets: 26 Procedures, 2 Actions, 6 Repos, 21 Stacks, 1 Sync; four new retry webhooks false and stored inline Actions match source.

Host /opt/homelab remains 0f522f0; #516's runner proposal on main is not applied there. No runner or Core/Periphery restart, ProcedureRun/ActionRun, workload deployment, build or workload source pull occurred; Core's normal Sync-cache fetch is not a host pull. Both pull timers are active, owned maintenance marker removed, no drain/pending relay intent; Core/Periphery retain 2026-10-25T11:33 starts. Workflow is disabled_manually. Native image check is now scheduled daily 03:00 America/Los_Angeles, with legacy auto-update schedule disabled; manual native/Action execution was not tested.

Historical preparation — 2026-10-27 05:25:28 UTC: phase was five_inputs_prepared_reddit_baseline_pending, bootstrap_ready=false. Approved preparation ran 05:24:58–05:25:28 UTC for oneuptime-incident-adapter, paperless-stirling-bridge, firefly, orders and gh-pr-dashboard: native root render/private HMAC/check all passed (exit 0), six rendered outputs including Orders OAuth JSON were byte-identical, and five 0700 state directories/0600 keys and fingerprints had valid 1001:1001 ownership. All five scoped SOPS/config/running-env credential gates passed, including actual Orders OAuth pinned mounts in both containers and fresh password-authenticated read-only PG15 adapter/PG18 Firefly sessions. No rotation, SQL writes, HBA reload or network token validation occurred.

LiteLLM check passed; Reddit check was stale before work and still failed at that capture. Both protected profiles' env/key/fingerprint bytes and metadata stayed unchanged. Root read-only HMAC RCA exactly matched Reddit's current clean source 3f0b0c9 plus pre-#514 Compose: only the gallery-worker read-only library change made it stale, not credential drift. That was scoped five-profile acceptance, not global readiness. Host 8bf7aa5 independently advanced to 0f522f0 (#514 Reddit/#515 Phoenix retained); the 645bc3c-pinned helper refused unexpected HEAD before fetch/merge, with no operator checkout write or rollback. Five-profile input code/configs matched 645bc3c..0f522f0; current 0f522f0 was used and four dirty tracked files preserved. Initial custom-auditor compatibility failures were corrected only in an ignored operator artifact, not host permissions or runtime configuration; they were not runtime failures.

Application/Core/Periphery IDs/start times stayed unchanged; no affected container was unhealthy and all healthcheck targets were healthy. Both pull timers were briefly paused under owned maintenance, then restored active and the marker removed. The runner was never drained/restarted, with no relay pending intent. During that preparation window, workflow stayed disabled; no Sync, grants, deployment or Core change occurred. Evidence remains root-private; see the historical preparation record.

Next owner action at that bootstrap capture (superseded): review the completion PR, then choose approval for the first scoped workflow deployment or optional image-publisher provisioning. App/Core PAT remain unprovisioned; manual GitHub check is available. First deployment/application smoke checks and unchanged retry remain pending. Do not repeat runner activation, preparation or bootstrap approval, or automatically enable dispatch; no scheduled general deployment. See bootstrap and least-privilege grants complete.

Rollback of the consumer must retain the new mount and use the existing explicit KOMODO_DEPLOY_ENV_FILE=/etc/komodo-deploy/komodo-deploy.env setting in reviewed persistent configuration. An old-mount-only rollback restores the bug; keep dispatch paused. Legacy credential cleanup requires separate explicit approval, never automatic migration cleanup. The Komodo workflow remains manual main dispatch only. Its homelab-production environment has no native reviewer protection, and main reports unprotected; do not infer a gate from that label or from this page's separate runner-maintenance environment policy.

The pinned runner image includes gnupg because maintained actions such as Codecov validate downloaded helper binaries with GPG. Treat missing action runtime dependencies as an image contract failure, not a reason to disable the action's integrity validation.

The pinned runner image includes libssl-dev and pkg-config because Rust crates that link OpenSSL natively (for example openssl-sys, pulled in transitively by reqwest's default native-tls backend) fail their build script without system OpenSSL headers, matching ubuntu-latest's toolchain.

The runner installation and job workspace under /runner use the container's ephemeral disk-backed writable layer. Do not mount /runner as tmpfs: package installs, checkouts, and build artifacts can otherwise exhaust the runner's memory cgroup even when process RSS is modest. The container is launched with --rm, so its writable layer is discarded after every job.

Exact-run quarantine: 2026-10-27¶

Applied 2026-10-27 10:49:47 UTC: exact-run job admission safely replaced the broad LAN drain. Orphan edge run 40469274207 remains queued with zero jobs and komodo-deploy.yml remains disabled_manually. No redispatch, cancel/rerun retry or deployment accompanied this restoration.

/etc/komodo-deploy/job-admission.sh is root:1001 0750, under the existing root:1001 0710 directory and same-path read-only directory bind mount. Guard SHA-256: a30f9450558ef734478141a6e9c459bc9af709a3b65de2f12e0e78275b62f632.
The instance environment is root:root 0600 and contains the literal, unquoted entry ACTIONS_RUNNER_HOOK_JOB_STARTED=/etc/komodo-deploy/job-admission.sh. The original environment is privately backed up; it contains a PAT. Never print the environment or backup, including during troubleshooting.
/etc/komodo-deploy/verify-job-admission.py is root:root 0600. Per-instance systemd drop-in /etc/systemd/system/github-runner@homelab-lan-01.service.d/90-orphan-run-guard.conf invokes ExecStartPre=/usr/bin/python3 -I -B /etc/komodo-deploy/verify-job-admission.py. The verifier fails closed on changed guard hash, modes, read-only mount/launcher contract or lost hook configuration. An explicit github_runner role apply may rewrite the instance environment; preserve/revalidate the hook, never bypass the verifier to start the service. Normal Ansible site.yml does not apply that role.
Tests passed in the real cached runner image with network none and default UID:GID 1001:1001: orphan run exit 1, another numeric homelab run exit 0, missing identity exit 2. The hook also rejects invalid or non-homelab identity. The verifier's positive check and changed-hash rejection passed. These are guard tests, not proof of GitHub registration, qualification or deployment acceptance.
Only the owned /var/lib/github-runner/drain/homelab-lan-01 marker was removed after tests, with its backup retained root-private. The actual LAN container is running as 1001:1001 plus group 10002, with the correct hook, read-only mount and no Docker socket; github-runner@homelab-lan-01.service is active. GitHub verifies homelab-lan-01 online.
Single non-deploying qualification run 40491873440, main 54ca4da: validation on homelab-ci-02, matrix on homelab-mgmt-01. GitHub API verified run and exact homelab-lan-01 job 122148472245 SUCCESS, including Set up runner (pre-job hook), checkout and Run Linux and macOS qualification checks. Normal LAN job execution is restored; no new test run was needed.
Both ansible-pull-check.timer and ansible-pull-apply.timer are active; /etc/homelab/MAINTENANCE, relay pending.json and the owned drain marker are absent. Private cutover-state.json and edge-window-20261027.json record guarded resume and qualification run 40491873440; main phase stays edge_dispatch_blocked_github_queue. No deployment, checkout reconciliation or mass restart is authorized by this checkpoint.

Retain the guard after restoration. Removal requires proof that the orphan is terminal and no executable pending work remains, followed by a deliberate reviewed follow-up. Do not remove the hook/verifier/drop-in merely because the service or a qualification succeeds. No automatic edge redispatch or workflow enablement follows. The temporary exact-homelab-ci-02 API route and management/deploy selectors are unchanged; watchdog remains a guarded mutation path, not blanket read-only work. Normal work may resume within these boundaries; do not replay unknown canceled work or mass-restart the fleet. The 10:53:25 UTC health snapshot has 49 running including the restored runner, 38 healthy, 11 without healthchecks, zero unhealthy/ restarting; not authenticated data acceptance. OneUptime and adapter October 27 B2 gaps are closed by explicitly owner-approved exact-snapshot copies; see the completed catch-up receipt. PR #520's local-retention fix is merged, but affected live units still lacked --retry-lock=5m at the 11:55 preflight. Normal convergence and next scheduled-run validation remain pending; no forced apply, restore/retention acceptance or broader mutation permission. Other health limitations remain separate.

Separate Gmail maintenance run 40489590509 placed validation/preview successfully; management-runner apply failed SSH No route to host to 172.29.240.13 / gha-docker-a while inspecting the drain marker for gmail-order-tracker-docker-01. Operation drain, changed=0, no mutation executed. An offline/unreachable on-demand host is plausible, not proof it was sleeping. Inspect the exact target before recovery; no blind replay or waking unrelated VMs.

API routing recovery: 2026-10-27¶

The edge-run containment hold also blocked the shared fleet entry point. Fleet run 40472345328 queued its Validate and summarize request job on LAN/deploy labels at 08:16 UTC, after the 08:14 hold, then was cancelled at 09:31 without a runner or executed steps. Its maintenance/watchdog routes never ran. Earlier fleet requests performed only an exact Gmail CI resume and a dry-run Gmail Docker drain—not a fleet-wide drain.

General placement was not globally broken: homelab security jobs ran at 09:17 UTC, and Gmail CI subsequently completed Python/maintenance jobs. Hyper-V workers being offline while idle is expected after the 900-second timeout. GitHub's public status reported operational; that does not resolve the individual edge orphan.

At 10:03:15 UTC, changed only RUNNER_FLEET_API_RUNS_ON from the LAN selector to the exact general selector above. Deployment and host-maintenance selectors, credentials, permissions, runner labels, host services and VM power settings were unchanged. The previous value is retained in the local private operation receipt artifacts/cutover-20261026/fleet-api-routing-recovery-20261027.json.

Proof: read-only health run 40485555074 completed 10:04:43 UTC on main 54ca4da. Both validation (17 seconds) and observation (22 seconds) actually ran on homelab-ci-02. The request was scoped to homelab, dry_run=true, authorize_disruptive=false; mutation/qualification routes were skipped. The report found 7 online, 2 offline, 9 discovered / 10 expected, 3 drifts. Successful collection is not a claim that every health finding is resolved, nor a reason to restart sleeping workers.

This is a temporary exact-instance recovery route, dependent on the general placement's scaler/wake and registration; it is not a new always-on/redundant control plane. Keep it until deliberate reassessment; the 10:49:47 UTC guarded LAN restoration does not itself authorize reverting the selector. Review registration/ qualification evidence and unresolved orphan risk before restoring the recorded selector or establishing a separately reviewed independent API lane; do not silently revert to the formerly blocked fallback. Previously cancelled requests are not automatically replayed: start with the intended operation's dry-run and exact scope.

The edge run remains queued with zero jobs; deployment is disabled. The earlier broad LAN hold is historical: the service is now active under exact-run quarantine, GitHub-online with non-deploying qualification successful and normal LAN job execution restored. Neither recovery authorizes another edge dispatch or direct Komodo execution. Retain the guard and normal active Ansible timers. See the incident record.

Central dispatcher operations¶

Open Actions → Runner fleet operations → Run workflow on the protected default branch (main). Every request is normalized before routing, and selectors resolve only against infra/github-runners/fleet.yml.

You do not SSH into Hyper-V VMs for normal revive, cancel, or maintenance. The dispatcher is the operator UI; SSH is last-resort when the Ansible adapter or management runner itself is broken.

Form fields¶

Field	What to put
`operation`	One of the operations in the table below
`fleet_target`	Preferred. Dropdown generated from `infra/github-runners/fleet.yml` (`placement:…`, `repository:…`, `runner:…`). Overrides `scope_type` / `scope_value` unless you leave the manual sentinel
`scope_type`	Manual fallback when `fleet_target` is the sentinel
`scope_value`	Manual fallback; empty for `fleet`
`platform`	Leave `all` unless isolating Linux vs Windows
`dry_run`	true for preview (default). false only when applying
`authorize_disruptive`	false for preview. true only with apply + a real reason
`drain_mode`	`graceful` unless owner explicitly authorizes `immediate`
`target_version`	Only for `upgrade`; must equal the manifest pin
`max_parallel`	Keep `1` until a canary succeeds
`reason`	Required for apply: 10–500 characters (incident or change note)

GitHub Actions cannot call the inventory API when rendering workflow_dispatch forms. Choices are therefore generated into the workflow YAML from fleet.yml by:

uv run python inventory/generators/render-runner-fleet-dispatch.py

CI Generator drift check fails if the dropdown or diagram drifts. After editing fleet.yml, regenerate and commit both outputs (pre-commit hook runner-fleet-generators also rewrites them when fleet.yml is staged):

uv run python inventory/generators/render-runner-fleet-dispatch.py
uv run python inventory/generators/render-runner-fleet-diagram.py

Placement map: runner-fleet-diagram.md.

Scope examples¶

Goal	Prefer `fleet_target`	Manual fallback
Whole fleet (read-only or watchdog preview)	sentinel + `scope_type=fleet`	`fleet` / empty
One repo's runners	`repository:dnd_session_parser`	`repository` / `dnd_session_parser`
One Hyper-V placement	`placement:general-a`	`placement` / `general-a`
Exact instance	`runner:dnd-session-parser-docker-01`	`runner` / exact name
Label intersection	sentinel + `labels`	`labels` / `docker` or `homelab,ci`

Exact runner names are case-sensitive and must match fleet.yml (for example tiktooker-docker-02, not Tiktooker-Docker-02).

Operations reference¶

Operation	Purpose	Mutation behavior
`inventory`	List expected and live runner identity and availability	Read-only
`health`	Report missing, offline, busy, and unexpected runners	Read-only
`activity`	Summarize recent run and job outcomes	Read-only
`queue`	Report queue pressure and oldest queued job age	Read-only
`report`	Produce the complete inventory, health, drift, queue, and activity report	Read-only
`drift`	Compare labels, platform, group, and manifest state	Read-only
`qualify`	Run identity, pinned-version, toolchain, temporary-file, and free-space checks on each exact selected runner	Workload probe only
`watchdog`	Find stale queued or in-progress workflow runs and optionally force-cancel them	Preview first; guarded API mutation
`maintenance`	Restore manifest-declared local service state where drift exists	Guarded host mutation
`cleanup`	Remove only an exact drained runner's container and diagnostics	Guarded host mutation
`upgrade`	Verify and apply the manifest-pinned runner version	Guarded host mutation
`restart`	Restart an exact drained and idle runner service	Guarded host mutation
`drain`	Stop new work and wait a bounded time for current work, or use explicitly authorized immediate mode	Guarded host mutation
`resume`	Remove the drain marker and resume the exact service	Guarded host mutation
`remediate`	Reconcile one exact drifted/offline runner; otherwise escalate	Guarded exact-target mutation

upgrade additionally requires target_version equal to the manifest-pinned version. Start with max_parallel: 1.

Run drain before maintenance, cleanup, restart, upgrade, or remediate; those operations fail closed without the exact drain marker. Draining an already-offline target is valid: it establishes the marker without interrupting work, after which exact-target remediation can reconcile the service. Finish with resume only after verification.

How to use matrix¶

What you see	First do (always dry-run)	Then apply when preview looks right	Do not
After a GitHub Actions outage: jobs stuck queued / in_progress	`watchdog` · `scope_type=fleet` · `dry_run=true`	Same scope · `dry_run=false` · `authorize_disruptive=true` · reason about the outage	Cancel healthy work while GitHub status is still red
Grafana shows offline / missing runners	`health` or `report` · `fleet` or one `repository`	See Revive dead runners below	SSH every VM first
Runner online in GitHub but never picks jobs (zombie session)	`health` + `drift` on that exact `runner`	`drain` → `remediate` (or `restart`) → `resume` on that exact name	Bulk `fleet` restart
Labels / class wrong vs `fleet.yml`	`drift` on `repository` or `runner`	`drain` → `remediate` → `resume`	Hand-edit labels in the GitHub UI
Local systemd/image drift suspected	`maintenance` dry-run on `placement` or `runner`	`drain` each target → apply `maintenance` → `resume`	Apply `maintenance` with `fleet` scope
Need confidence a worker is healthy	`qualify` on one exact `homelab` runner first	Expand place/repo only after canary passes	Cross-repo `qualify` from `homelab` (fails closed)
Disk / toolchain / version check	`qualify` exact `runner`	Fix host or run `upgrade` after drain	Skip the canary
Queue age high but runners look idle	`queue` + `health`	Often label mismatch or repo-scoped idle capacity elsewhere — fix matching, do not add random runners	Assume “idle capacity” is fungible across repos
One bad container / leftover diagnostics	`drain` exact runner	`cleanup` → `resume`	`cleanup` without drain

Default thresholds for watchdog: queued ≥ 30 minutes, in-progress ≥ 360 minutes. Override only with a reviewed variable change.

Revive dead / sleeping / zombie runners¶

Use this when five (or any) runners are offline, missing, or “listening” but not taking work. Prefer one exact runner at a time.

flowchart TD
  A["health or report"] --> B{"Offline / missing / zombie?"}
  B -->|no| Z["Stop — no host mutation"]
  B -->|yes| C["Pick exact runner name from report"]
  C --> D["drain dry-run then apply"]
  D --> E["remediate dry-run then apply"]
  E --> F["health + drift on that runner"]
  F --> G{"Healthy?"}
  G -->|yes| H["resume dry-run then apply"]
  G -->|no| I["Escalate: adapter/SSH or stale GitHub registration ID"]
  H --> J["Next dead runner"]

Inventory the damage (read-only)
operation=health or report
scope_type=fleet, scope_value empty
dry_run=true, authorize_disruptive=false Download report.md. Note exact runner names that are offline, missing, or unexpected.
Optional: clear stuck workflow runs first If GitHub just recovered from an outage and many jobs are ancient queued / in_progress, run watchdog preview then apply (see matrix). Stale jobs can look like dead runners when the real issue is a zombie registration on the job side.
Drain one exact runner (safe even if already offline)

Field	Value
`operation`	`drain`
`scope_type`	`runner`
`scope_value`	e.g. `dnd-session-parser-docker-01`
`drain_mode`	`graceful`
`dry_run`	`true` first, then `false`
`authorize_disruptive`	`true` only on apply
`reason`	e.g. `Revive offline runner after Actions outage`

Remediate that same exact runner Same scope as drain. operation=remediate. Preview (dry_run=true) must show a non-empty plan. Apply with authorize_disruptive=true and the same reason style.
Prefer remediate for offline / missing / label drift.
Use restart only when the runner is drained, idle, and you only need a service bounce.
Applied remediate accepts one exact runner only.
Verify
health and drift with scope_type=runner and the same name.
Confirm online in the report (and in Grafana if you use it).
For homelab runners only, optional qualify canary.
Resume
operation=resume, same exact runner scope, preview then apply.
Resume removes the drain marker so jobs can match again.
Repeat for each remaining dead name. Do not widen to fleet for apply.

If remediate's plan is empty but the host is still dead, the API does not see actionable drift — escalate: Ansible/management-runner path, or a stale server-side registration that needs manual exact-ID cleanup (never bulk delete). See Failure handling.

Concise operator procedure¶

Run report for the intended scope and inspect the summary and artifact.
Run qualify on one exact runner. Expand to one placement group, then one repository, only after the canary passes.
For maintenance, select the narrowest exact scope, leave dry_run: true, keep authorize_disruptive: false, and provide the intended reason.
Review the generated plan, request fingerprint, target identities, current IDs/status, and action count. A zero-action remediate plan means no API-visible drift matched the selector.
To apply, rerun with the same exact scope, dry_run: false, authorize_disruptive: true, and a specific incident/change reason. The owner reviews the protected-environment approval.
Run health, drift, and qualify again. Confirm queue age, labels, service state, and the applied report before expanding scope.

OneUptime Stage 6 timeline notes¶

When a fleet dispatch is part of an active OneUptime incident:

Acknowledge the incident in OneUptime.
Prefer exact-runner health / drift / report with dry_run: true.
Record the GitHub Actions run on the incident via scripts/oneuptime_post_remediation_note.py or workflow oneuptime-remediation-note.yml (environment oneuptime-remediation).
Only then preview/apply disruptive operations with protected approval.

Runner fleet Prometheus alerts include action_url pointing at runner-fleet-dispatch.yml. Cross-repo qualify still fails closed from homelab (see Qualify topology limit).

Never use a fleet selector merely for convenience. The requested scope is an authorization boundary as well as a filter.

maintenance deliberately plans every selected target because GitHub's API cannot see local systemd/image drift. It remains drain-gated and serial. remediate is narrower and plans only missing, offline, label-drifted, or group-drifted targets.

Safety model¶

Dry-run is the default. An applied mutation requires all of:

dry_run: false;
the explicit boolean authorize_disruptive: true;
a reason between 10 and 500 characters;
an exact, bounded selector that resolves to configured targets;
a fresh preview whose request and plan fingerprints still match;
a fresh GitHub state recheck immediately before each action;
approval through runner-fleet-maintenance.

Applied host maintenance rejects fleet scope. Use an exact runner, repository, placement, label set, or declared organization group so a mistake cannot restart every configured worker.

Plans expire after 30 minutes by default. Changed runner ID, name, status, scope, plan fingerprint, or request fingerprint fails closed. Applied fleet mutations share one non-cancelling concurrency group, so two repair operations cannot race.

There is no runner deletion or deregistration operation. remediate accepts only one exact runner when applied, rechecks the same ID/name/status, and invokes the topology adapter to reconcile local service state. A stale server-side registration that still requires deletion is escalated for manual exact-ID recovery; never bypass this with bulk gh api DELETE.

The Linux systemd/Docker and Windows local-service adapters validate exact service/container/install paths, require drained and idle state for destructive cleanup or restart, and record idempotency results. Immediate drain requires separate immediate authorization. The Ansible adapter uses argument arrays, allowlisted executables, exact inventory host mapping, and a one-runner playbook allowlist.

For repository runners, drain first removes all custom scheduling labels from the freshly rechecked runner ID. Existing work can finish, but homelab jobs can no longer match because every workflow requires repository/class labels. The adapter then waits for Runner.Worker and stops the exact service. If adapter execution fails, the control plane attempts to restore the manifest labels. Do not introduce workflows that target only self-hosted, OS, or architecture labels; they would bypass this quarantine. Organization-runner drain fails closed until a topology adapter implements runner-group quarantine.

Watchdog¶

The watchdog treats a queued run as stale after 30 minutes and an in-progress run as stale after 360 minutes unless reviewed overrides are active. Preview records the run ID, repository, status, timestamp, age, and a precondition fingerprint.

Apply downloads that exact preview, verifies its signature and age, restricts each action to the original repository selector, refetches the run, and checks that status and timestamp are unchanged and still over threshold. Only then does it call GitHub's force-cancel endpoint. A completed, updated, or otherwise changed run is reported as failed and is not cancelled.

If GitHub is experiencing an Actions incident, leave the watchdog in preview or withhold maintenance approval. Do not repeatedly cancel and restart healthy work during a control-plane outage.

Reports and alerts¶

Every operation writes a GitHub step summary and uploads redacted artifacts for 30 days:

report.json for structured automation;
report.csv when runner rows are present;
report.md for operator review;
metrics.prom for Prometheus ingestion;
audit.jsonl for append-only event review;
plan.json for watchdog and maintenance previews.

Artifact names include the operation and workflow run identity. Secret-like keys and token patterns are redacted before reports are written. Preserve a relevant artifact outside the 30-day window when it is incident or change evidence.

Observation reports missing/offline runners, label or group drift, unexpected records, busy/online utilization, queue age, recent run/job outcomes, and API retry diagnostics. Qualification and apply failures fail their jobs, providing the normal GitHub Actions notification signal. The scheduled observation also opens or updates one runner-fleet-alert issue for failed collection, offline or unexpected runners, drift, or queue age at the configured threshold (bodies include runner names and drift issue codes, not only counts); it closes that issue after recovery. metrics.prom is portable report output; it is not automatically scraped until an operator or collector publishes it. Do not claim Alertmanager coverage from artifact generation alone.

Rollout, canaries, and rollback¶

Use this rollout order:

Observe the fleet.
Qualify one idle general runner in general-a.
Qualify one Docker runner without changing its socket boundary.
Qualify the LAN runner last.
Preview maintenance on one non-management runner.
Apply to that canary with max_parallel: 1, re-observe, then expand by placement group.

For an upgrade, drain the canary gracefully, apply only the manifest-pinned version, image tag, and release SHA-256, qualify it, and resume it. The build records version/checksum provenance labels and the adapter executes the image's Runner.Listener --version before success. Keep at least one healthy runner in each required class while rolling.

Rollback means stopping scope expansion and returning the exact canary to the last manifest-pinned configuration through Ansible. If the runner is merely drained, use resume; idempotent replay makes repeated resume safe. If a maintenance or upgrade fails, keep the target drained until host state and the report are understood. Never roll back by copying another runner's configured directory or by deleting registrations in bulk.

Ansible from CAPTAINKANGAPOO¶

Hyper-V NAT guests (172.29.240.11–.16) are reachable from the Hyper-V host, not from WSL's default network namespace and not from infra-services. Do not run ansible-playbook inside a runner guest (no ansible, no repo checkout). Do not rely on native Windows ansible-core either — it often dies at startup with OSError: [WinError 87] on stdin.

Proven control path (2026-08):

SSH key for runneradmin: ~/.ssh/github-runner-vm-admin (see ~/.ssh/gha-runner-ssh.cfg). Prefer including that snippet from ~/.ssh/config, or pass -i / -F explicitly.
Run ansible inside WSL, but tunnel TCP through Windows OpenSSH so the Hyper-V NAT route works.
Force Windows ssh.exe to ignore the user's config (-F NUL). Otherwise Include ~/.ssh/1Password/config fails non-interactively (Permission denied → terminating, 1 bad configuration options) and mid-playbook forks go UNREACHABLE.
Prefer forks = 1 and ControlMaster=no for long multi-host rolls from this path; parallel forks amplify the broken-ProxyCommand failure mode.

Example WSL SSH config fragment (~/.ssh/gha-via-windows.cfg):

Host gha-general-a 172.29.240.11
  HostName 172.29.240.11
  User runneradmin
  IdentityFile ~/.ssh/github-runner-vm-admin
  IdentitiesOnly yes
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  ProxyCommand /mnt/c/Windows/System32/OpenSSH/ssh.exe -F NUL -i C:/Users/renjamin/.ssh/github-runner-vm-admin -o IdentitiesOnly=yes -o IdentityAgent=none -o BatchMode=yes -o StrictHostKeyChecking=accept-new -W %h:%p runneradmin@%h

Use a throwaway ANSIBLE_CONFIG without ara_default if WSL ansible lacks ARA. Point ssh_args at the absolute config path (-F /home/someone/.ssh/gha-via-windows.cfg). Ansible passes ANSIBLE_SSH_ARGS to ssh without shell expansion, so a literal ~ fails with Can't open user config file ~/.ssh/gha-via-windows.cfg. Include a Host block for each runner (gha-canary-a / 172.29.240.16 is required for the OneUptime wake canary).

After playbook success, confirm registration in journal (Listening for Jobs) and in the target repo's Actions → Runners list. Transient Runner connect error: Conflict after --replace usually clears within a minute; persistent curl: (22) ... 404 means the host registration PAT still lacks that repository.

Failure handling¶

Symptom	Action
App token creation fails	Verify App installation, exact permission level, App ID variable, and the environment private-key secret; do not substitute the host PAT
Selector matches nothing	Compare case-sensitive runner identity and manifest repository/placement group; do not widen to fleet
Plan is stale or fingerprints differ	Create a new preview and repeat approval
Fresh-state check fails	Re-observe; another actor or GitHub changed the target
Adapter is unavailable	Verify the management runner, execution mode, Ansible inputs, SSH path, and `ansible@v1`; no host mutation occurred
Graceful drain times out	Inspect the exact job; extend a bounded timeout or obtain owner authorization for immediate drain
Qualification fails disk check	Free disk space (docker-class hosts: Docker GC); qualification requires at least 20% free
API retries exhaust	Check GitHub status and rate limits; wait rather than bypassing controls
GitHub cancel / force-cancel returns HTTP 500	Preserve the run URL and GitHub request ID, then open a GitHub Support case; matching online runners cannot clear the orphaned run
Runner has a stale server session	Drain and remediate one exact runner after GitHub recovers; deregistration remains a manual exact-ID escalation
Registration curl 404	Add the target repo to the SOPS registration PAT allowlist; restart `github-runner@<name>`
Ansible UNREACHABLE via WSL proxy	Ensure ProxyCommand uses `ssh.exe -F NUL`; do not load the Windows user `Include`/1Password config
Allowlist assert fails	Pass only instance names that exist on the limited host (`allowlist ⊆ github_runner_instances`)

Local host health¶

Each host runs github-runner-host-health.timer every 15 minutes. It uses the SOPS-protected registration credential to check remote online state and exact labels, then checks local systemd units, runner images, stale containers, 80% disk pressure, and recent cgroup or system OOM kills. No administrative credential is stored in workflow secrets.

Inspect Linux failures with:

systemctl status github-runner-host-health.service
journalctl -u github-runner-host-health.service

A systemd restart obtains a fresh registration token and starts a clean ephemeral container. Unit shutdown stops and removes only that runner's container; startup removes an orphaned same-name container before registration. Never copy a configured runner directory between instances.

Docker GC on docker-class hosts¶

gha-docker-a / gha-docker-b share the host Docker engine with CI jobs. Ephemeral runner containers use --rm, but compose volumes, dangling layers, and BuildKit cache stay on / and filled those guests (100% / 83% before a manual tidy). Host health only alerts at 80% used.

github-runner-docker-gc.timer runs on hosts that allocate docker-class runners and no LAN runner. It never enables on infra-services. The script also refuses to run when /opt/homelab/services exists or any instance is RUNNER_CLASS=lan.

The Hyper-V scaler powers these guests off after ~15 minutes idle. OnUnitActiveSec only advances while the VM is up, so a 6h relative timer could starve across short wake cycles. The timer therefore uses:

OnBootSec=2min — GC soon after each scaler start (inside the idle window)
OnCalendar=hourly with Persistent=true — catch up after long offline gaps

flowchart TD
  scaler["Hyper-V scaler Start-VM"] --> boot["guest boot"]
  boot --> onboot["OnBootSec 2min GC"]
  job["docker-class CI job"] --> sock["host Docker socket"]
  sock --> leftovers["dangling volumes / orphan BuildKit / build cache"]
  leftovers --> gc["github-runner-docker-gc.timer"]
  onboot --> gc
  calendar["OnCalendar hourly Persistent"] --> gc
  gc --> keep["keep named CI cache and registered builders"]
  gc --> free["reclaim dangling + orphan BuildKit state"]

Each run:

docker container prune for stopped containers older than 12 hours
Remove orphan buildx_buildkit_* containers/volumes that are no longer registered in docker buildx ls (setup-buildx-action / Copa leftovers)
docker volume prune for dangling volumes only
docker image prune for dangling layers only — not -a, so tags such as tiktok-backend:ci, Playwright, and Copa patched images stay as CI cache
docker buildx prune --all --reserved-space 4gb for BuildKit cache older than 24 hours

Inspect with:

systemctl status github-runner-docker-gc.timer
journalctl -u github-runner-docker-gc.service
cat /var/lib/node_exporter/textfile_collector/github_runner_docker_gc.prom