Skip to content

ADR-004: Org-scoped shared runner pool

  • Status: Proposed (owner gates pending)
  • Date: 2026-11-03

Context

Every general and docker runner in infra/github-runners/fleet.yml is registered to one repository. Today that is 25 general runners on gha-general-a/gha-general-b and 18 docker runners on gha-docker-a/gha-docker-b, split across 12 repositories. A busy repository queues while idle runners registered to other repositories sit unused.

The spadoople organization is on GitHub Free. Free organizations have one runner group, Default. They cannot create other groups or restrict a group to specific workflows. The organization also owns 7 public repositories (Sandbox, ins_loader, OF-Scraper, OF-Scraper-Docs, Civicomfy, comfyui-ollama, ComfyUI-MiniMaxH3-Prompt-Writer). All 12 fleet repositories are private (verified 2026-11-03).

The runner model already accepts scope.type: organization and group. The registration, scaler, fleet-operation, host-health, alert and dashboard paths still assume repository scope. Changing those paths is phases 1 to 4 of the rollout plan. They are not part of this decision record.

Decision

  1. Register the general and docker classes as organization-scoped runners in the Default runner group. Name them pool-general-{a,b}-NN and pool-docker-{a,b}-NN, with labels self-hosted,Linux,X64,pool,<class>,<exact-name>. They get no repository label.
  2. The pool serves only private repositories. The Default group must have allows_public_repositories: false (owner gate A).
  3. Isolation inside the pool is by label only. Repositories opt in through a top-level pool_repositories list in fleet.yml and by setting CI_RUNS_ON / DOCKER_RUNS_ON to the pool labels.
  4. These runners stay repository-scoped to homelab. With only the Default group available, repository scope is the only control that keeps other repositories off them:
  5. homelab-lan-01: Komodo credential mount, /opt/homelab checkout and the exact-run admission hook.
  6. homelab-mgmt-01: runs Ansible with fleet keys.
  7. homelab-ephemeral-canary-01: the OneUptime exact-label wake proof.
  8. Size the pool from measured concurrency (below), not from the current per-repository runner counts.

Accepted risks

  • Any private org repository can target pool labels. This is acceptable only because every private repository is owner-controlled. A private repository with an outside collaborator, or a public repository allowed into the Default group, voids this assumption. Revisit the decision if either happens.
  • The docker pool shares each guest's Docker daemon, images, volumes and build cache across repositories. That is already true per VM today, where several repositories' docker runners share gha-docker-a and gha-docker-b.
  • Secrets stay protected by ephemerality, not by group isolation. Pool runners are --ephemeral containers, and a job's secrets do not survive into the next job's runner. Job containers on the docker class can still see shared daemon state (images, volumes, cache). Do not put credentials in build cache or named volumes.
  • The org registration credential is broader than today's per-repository PAT. It can register a runner for any repository in the organization. Keep it in SOPS, only on runner hosts, and separate from the existing PAT (owner gate B).
  • Workflow restriction is unavailable on Free. Any workflow in an allowed repository can select the pool. This is the same as today for each repository's own runners.

Sizing evidence

Read-only measurement from the GitHub Actions jobs API, 2026-10-20 08:40 UTC to 2026-11-03 08:40 UTC (14 days). Coverage was full, not sampled: 3,032 workflow runs across the 12 fleet repositories, with every job fetched (filter=all, about 3,100 API calls).

Method:

  • Class is taken from the job's requested labels. Docker means the labels include docker. General means ci without any of docker, lan, deploy, runner-management or runner-remediation-canary. LAN, management, canary and GitHub-hosted jobs are excluded.
  • Only jobs that started on a runner count. Jobs canceled while still queued are excluded.
  • 280 jobs were excluded because GitHub copies the original timestamps into a re-run attempt for jobs that were not re-run. Those jobs have started_at < created_at, and counting them would double-count concurrency.
  • "Running" is the exact interval sweep over started_at..completed_at. Percentiles are time-weighted over the time when at least one job of that class was running. Minute buckets were rejected because general jobs are shorter than a minute at p50, so buckets overcount.
  • "Demand" means queued plus running (created_at..completed_at). It is an upper bound on what a pool would have absorbed.
  • Queue wait is started_at - created_at. It includes any Hyper-V on-demand wake time.
Metric General Docker
Jobs started 12,390 1,928
Running peak 10 (2026-10-21 05:21 UTC) 6 (2026-11-02 23:23 UTC)
Running p50 / p90 / p95 / p99 2 / 5 / 6 / 7 1 / 2 / 2 / 3
Busy hours 43.7 60.5
Demand peak 64 (all homelab, 6 running) 26 (all telegram_companion, 0 running)
Demand p95 / p99 22 / 37 8 / 16
Queue wait p50 / p90 / p99 41 s / 152 s / 380 s 136 s / 743 s / 2,112 s
Queue wait max 1,989 s 3,990 s
Job duration p50 / p90 / p99 19 s / 67 s / 263 s 66 s / 410 s / 841 s

Reading the numbers:

  • The running peaks are supply-limited by per-repository partitioning. homelab alone reached 8 concurrent general jobs, because it has 6 dedicated general runners plus bursts.
  • The demand peaks are single-repository bursts. The docker peak had zero jobs running, so it reflects unavailable capacity (guests asleep or down), not concurrency.
  • Docker queue waits are long at p90. That is consistent with per-repository docker runners being 1 to 3 deep and with on-demand wake.
VM Pool runners Peak + headroom Memory ceiling at 4 GiB per runner Guest max
gha-general-a 8 general peak 10 → 16 total (1.6×) 32 GiB 4 vCPU / 28 GiB
gha-general-b 8 32 GiB 4 vCPU / 28 GiB
gha-docker-a 4 docker peak 6 → 8 total (1.33×) 16 GiB 6 vCPU / 16 GiB
gha-docker-b 4 16 GiB 6 vCPU / 16 GiB

Rationale:

  • General, 16 total. The measured peak is 10. The pool also covers demand p90 (19) closely. Short jobs (p90 67 s) drain a burst quickly. The 32 GiB memory ceiling per VM is a mild overcommit against 28 GiB. Container limits are ceilings, not reservations, and the 99th-percentile running concurrency across both VMs is 7. Today's ceilings are much higher (13 and 12 runners at 4 GiB, or 52 and 48 GiB).
  • Docker, 8 total. The measured peak is 6, and 8 matches demand p95. Service containers, such as Reddit Explorer's PostgreSQL, run on the guest daemon outside the runner container's limit. That argues against more than 4 per 16 GiB guest. The current inventory sets github_runner_memory_limit: 6g on docker hosts, which gives a 24 GiB ceiling at 4 runners. Choose 4 GiB or 6 GiB in the manifest phase. Either way it is far below today's 9 runners at 6 GiB (54 GiB) per guest.
  • One-VM loss. Either remaining VM of a class still covers that class's running p99 (general 8 ≥ 7, docker 4 ≥ 3).
  • Re-measure after cutover. Pool concurrency is no longer supply-limited per repository. Re-run this measurement two weeks after phase 5, and adjust counts and target_cap in phase 6.

Owner gates

These must pass before any org-scoped runner registers (plan phase 1 rollout). The agent token has repo, read:org and workflow scopes. orgs/spadoople/actions/runners and orgs/spadoople/actions/runner-groups return HTTP 403 with that token, so the agent cannot read or verify any of these.

A. Default group excludes public repositories

The organization has 7 public repositories. If the Default group allows public repositories, fork pull requests on those repositories could run on pool hardware. This gate blocks rollout.

  1. gh auth refresh -h github.com -s admin:org
  2. List the runner groups:
gh api orgs/spadoople/actions/runner-groups \
  --jq '.runner_groups[] | {id, name, default, visibility,
    allows_public_repositories, restricted_to_workflows}'
  1. The expected result is name: "Default", default: true and allows_public_repositories: false.
  2. If the value is true, go to Organization settings → Actions → Runner groups → Default and clear Allow public repositories. Then repeat step
  3. Leave workflow restriction off; Free cannot set it.

B. Org registration credential (new SOPS key)

  1. Create a fine-grained PAT with resource owner spadoople and Organization permissions → Self-hosted runners: Read and write. Give it no repository permissions. Alternatively, add the same organization permission to the Operator App.
  2. Store it as a new key, github_runner_org_registration_pat, in infra/ansible/inventory/group_vars/github_runner_hosts/registration.sops.yaml next to github_runner_registration_pat. Do not widen or replace the existing per-repository PAT, which LAN, management and canary keep using.
  3. Verify read access without registering anything. Paste the token at the prompt; do not put it in shell history:
$env:GH_TOKEN = Read-Host -MaskInput 'org runner PAT'
gh api orgs/spadoople/actions/runners --jq '.total_count'
Remove-Item Env:GH_TOKEN

Expect a number, not HTTP 403. Record the token's expiry date for rotation.

C. Scaler and collector read org runners

The Hyper-V scaler uses a copy of the github-actions-collector App PEM (see github-actions-runners.md). One App permission change therefore covers both.

  1. In the App settings, set Organization permissions → Self-hosted runners: Read. Then accept the updated permissions on the spadoople installation.
  2. Verify with gh api orgs/spadoople/installations --jq '.installations[] | {app_slug, permissions: .permissions.organization_self_hosted_runners}'. This needs the admin:org refresh from gate A. Expect "read" for the collector App.
  3. On infra-services, check that the collector logs no longer show HTTP 403 on OrgRunners: docker compose logs --since 15m github-actions-collector.

Owner checklist

  • [ ] A: Default group allows_public_repositories: false, verified by API
  • [ ] B: org PAT created with Self-hosted runners read/write, stored as github_runner_org_registration_pat, read check returns a count, expiry recorded
  • [ ] C: collector/scaler App has org Self-hosted runners Read, installation accepted, no OrgRunners 403
  • [ ] Accept the risks above, and change this ADR's status to Accepted

Consequences

Positive

  • Idle general and docker capacity becomes fungible across fleet repositories.
  • The fleet shrinks from 43 to 24 general and docker runners. Queue waits are expected to fall, mainly for docker.
  • A repository is added to the pool by one list entry plus repository variables, instead of new runner instances.

Negative

  • Silent failure modes appear until phase 4 lands: dashboards filtered by repository, the NoIdleCapacity join, and on-demand idle suppression. Phase 4 must merge before cutover.
  • The r7 image rollout restarts every runner, including LAN behind its admission guard.
  • Organization-wide runner names must be unique.

Revisit conditions

Reconsider this decision if:

  • the organization adds a private repository with outside collaborators;
  • the organization moves to a plan with custom runner groups or workflow restriction (then move the pool into a dedicated group);
  • the Default group must allow public repositories for another reason;
  • measured pool concurrency after cutover exceeds the per-guest limits above.