Skip to content

GitHub Actions runner fleet

The fleet contains ten repository-scoped Linux/X64 runner instances. General workers are ephemeral unprivileged containers. Docker workers are placed on isolated build VMs and are the only class with the Docker socket. The homelab LAN worker has host networking, a read-only operational configuration mount, and a read-write mount of the live /opt/homelab checkout. It never receives dev or dependabot labels.

No self-hosted runner receives the special dependabot label. GitHub-generated Dependabot update jobs require Docker access, so they remain on GitHub-hosted infrastructure. CI for Dependabot pull requests still uses the repository's ordinary ci runners. Keep Settings > Advanced Security > Dependabot on self-hosted runners disabled in every repository.

Placement

The four CI/build placement groups run as isolated Hyper-V VMs on the always-on CAPTAINKANGAPOO workstation. The LAN runner remains on infra-services. Host-specific github_runner_instances in infra/ansible/inventory/github-runners.yml contain only the entries assigned to each host.

Registration credential

Create infra/ansible/inventory/group_vars/github_runner_hosts/registration.sops.yaml from the adjacent example and encrypt it with SOPS before committing or deploying. The value is never a workflow secret and must not appear in normal inventory, logs, or shell history. Use a fine-grained PAT restricted to runner administration on the five target repositories.

Deploy

  1. Build the isolated general and Docker VMs and confirm the LAN host is reachable.
  2. Review infra/ansible/inventory/github-runners.yml and confirm its host addresses and instance allocation match the deployed fleet.
  3. Create and SOPS-encrypt infra/ansible/inventory/group_vars/github_runner_hosts/registration.sops.yaml from the adjacent example. Commit only the ciphertext.
  4. Run the playbook with both inventory sources:
ansible-playbook \
  -i infra/ansible/inventory/generated.yml \
  -i infra/ansible/inventory/github-runners.yml \
  infra/ansible/playbooks/github-runners.yml
  1. Set repository variables:
  2. CI_RUNS_ON=["self-hosted","Linux","X64","<repo>","ci"]
  3. DOCKER_RUNS_ON=["self-hosted","Linux","X64","<repo>","docker"]
  4. DEPLOY_RUNS_ON=["self-hosted","Linux","X64","homelab","lan","deploy"]
  5. Dispatch runner qualification before moving CI workloads.

The LAN container runs as lanrunner UID/GID 1001, matching the owner of the live checkout and Komodo credential source. Its launcher mounts /opt/homelab read-write, reapplies a read-only ACL for UID 1001 to /etc/homelab/labctl.env, and deliberately fails before registration if the checkout ownership or Komodo credential readability is wrong. Do not make either credential source world-readable. General and Docker runners continue to run as UID/GID 10001 and never receive the live checkout mount.

Health and recovery

Each repository's scheduled health workflow targets every uniquely labeled runner and checks version 2.335.0, class isolation, Docker readiness where applicable, and clean ephemeral startup. Each host also runs github-runner-host-health.timer every 15 minutes. It uses the SOPS-protected registration credential to check remote online state and exact labels, then checks local systemd units, runner images, stale containers, and 80% disk pressure. No administrative credential is stored in workflow secrets. Inspect failures with systemctl status github-runner-host-health.service and journalctl -u github-runner-host-health.service.

A systemd restart creates a fresh registration token and clean ephemeral container. Unit shutdown stops the runner container gracefully and then removes any remainder; startup also removes an orphaned same-name container before registration. For recovery, stop the affected unit, revoke the runner in repository settings, clear only that instance's container/workspace, and restart the unit. Never copy a configured runner directory between instances.

Recover a stale GitHub session

After a GitHub Actions outage, a locally healthy runner can remain offline while its container repeatedly logs either A session for this runner already exists or Runner connect error: Error: Conflict. A normal restart cannot clear this server-side session conflict because registration replacement and runner-session cleanup are separate operations.

Recover only the affected instance. Do not bulk-delete runner records and do not automate deletion from the host health check.

  1. On an authenticated administration workstation, resolve the exact runner record and confirm that GitHub reports it offline:
owner=notarealemail
repo=tiktooker
runner_name=tiktooker-ci-01

gh api "repos/${owner}/${repo}/actions/runners" \
  --jq ".runners[] | select(.name == \"${runner_name}\") | [.id, .name, .status] | @tsv"
  1. On the runner host, stop only that instance and confirm its container is no longer running. The unit can temporarily display failed because stopping terminates its blocking docker run process.
runner_name=tiktooker-ci-01

sudo systemctl stop "github-runner@${runner_name}.service"
sudo docker ps --filter "name=^/gha-${runner_name}$"
  1. Copy the numeric ID from step 1. Re-query that exact record immediately before deletion and require it to still be the expected name and offline:
owner=notarealemail
repo=tiktooker
runner_id=REPLACE_WITH_VERIFIED_NUMERIC_ID

gh api "repos/${owner}/${repo}/actions/runners/${runner_id}" \
  --jq '[.id, .name, .status] | @tsv'
# Continue only when this prints the expected ID, name, and offline status.
gh api --method DELETE \
  "repos/${owner}/${repo}/actions/runners/${runner_id}"
  1. Start the stopped instance. It obtains a new registration token and creates a fresh repository-scoped runner record:
runner_name=tiktooker-ci-01

sudo systemctl reset-failed "github-runner@${runner_name}.service"
sudo systemctl start "github-runner@${runner_name}.service"
sudo systemctl is-active "github-runner@${runner_name}.service"
  1. From the administration workstation, confirm the runner is online with its exact labels:
owner=notarealemail
repo=tiktooker
runner_name=tiktooker-ci-01

gh api "repos/${owner}/${repo}/actions/runners" \
  --jq ".runners[] | select(.name == \"${runner_name}\") | {id, name, status, labels: [.labels[].name]}"
  1. On the runner host, run the host health check:
sudo systemctl start github-runner-host-health.service
sudo systemctl show github-runner-host-health.service -p Result --value

Expected recovery result: the runner is online, the health service reports success, and queued jobs begin dispatching. If GitHub is still reporting an active Actions incident, stop here and wait rather than repeatedly replacing the record.