Skip to content

2026-11-03 — Org runner pool

TL;DR

The organization-scoped runner pool from ADR-004 is specified in source: #587–#590 are merged, and the manifest (#591) plus a dry-run cutover report (#592) are open. Nothing is registered, and merging the manifest does not authorize a pool runner. Registration stays blocked until the Default group policy is confirmed (unread; the runner-group GET returned HTTP 403), the separate org PAT is in SOPS, the collector App can read org runners, and image r7 is already running on every runner host.

Phase status

Phase Scope State
0 ADR-004, sizing, owner gates PR #590 merged; owner gates not verified
1 Org registration path, image 2.338.2-r7, host health PR #587 merged; r7 not rolled out; image guard stacked into #591
2 pool_repositories and 24 pool instances in the manifest PR #591 open; merge does not register runners
3 Scaler, observe, watchdog, drain, qualify, generators PR #588 merged
4 Collector scope labels, pool NoIdleCapacity, dashboard PR #589 merged
5 Per-repository cutover in batches Not started; PR #592 is a dry-run only
6 Retire repository-scoped general and docker runners Not started; destructive, owner-approved
7 Runbook, ADR, journal ADR merged in #590; runbook and this entry are in #591

Pull requests 587 through 590 are on main. Pull requests 591 and 592 are open. No repository variable has changed.

What shipped (in source)

  • Registration (#587, merged). entrypoint.sh branches on RUNNER_SCOPE to use the org registration-token endpoint, --url https://github.com/spadoople and --runnergroup Default. instance.env.j2 renders scope, group and the per-scope credential. The role requires a separate SOPS key, github_runner_org_registration_pat. A later commit on that branch, stacked into #591, refuses to build or write env for a selected organization instance unless the image already on the host is r7 and its entrypoint requests an organization registration token. Image r6 ignores RUNNER_SCOPE. The missing, empty, or reused per-repository PAT still fails closed before that check. runner-host-health lists org runners and labels them scope="organization".
  • Control plane (#588, merged). The fleet loader accepts pool_repositories. The scaler lists /orgs/spadoople/actions/runners once and polls pool repositories for jobs. Idle-offline suppression, observe, watchdog, drain quarantine and qualify cover org runners. Generators emit group:Default and pool:<class> dispatcher scopes, a pool node in the diagram, and scope plus pool_repositories in the scaler map.
  • Observability (#589, merged). Collector job metrics carry the real scope. GitHubRunnerNoIdleCapacity has a pool branch keyed on class and placement group. The fleet overview dashboard gains a scope variable and keeps pool runners visible under $repository.
  • ADR (#590, merged). ADR-004 records the Free-plan constraint, accepted label-only isolation, and sizing from a 14-day measurement: general running peak 10, docker running peak 6.
  • Manifest (#591).
  • pool_repositories lists the 12 repositories in fleet.yml. The plan said 13; the manifest has 12.
  • 8 general pool runners each on gha-general-a and gha-general-b, and 4 docker pool runners each on gha-docker-a and gha-docker-b, mirrored in the Ansible inventory.
  • The 46 repository-scoped runners stay for the overlap, so target_cap rises from 48 to 72 for 70 instances.
  • The runbook pool section covers labels, opt-in, selectors, cutover, rollback, retirement and the owner gates.
flowchart LR
  subgraph repos [12 pool_repositories, private]
    ci["CI_RUNS_ON: pool, ci"]
    dk["DOCKER_RUNS_ON: pool, docker"]
  end
  subgraph defaultGroup [Org Default runner group]
    ga["pool-general-a-01..08"]
    gb["pool-general-b-01..08"]
    da["pool-docker-a-01..04"]
    db["pool-docker-b-01..04"]
  end
  subgraph repoScoped [Repository-scoped to homelab]
    lan[homelab-lan-01]
    mgmt[homelab-mgmt-01]
    canary[homelab-ephemeral-canary-01]
  end
  ci --> ga
  ci --> gb
  dk --> da
  dk --> db
  scaler["Hyper-V scaler polls pool_repositories"] --> ga
  scaler --> gb
  scaler --> da
  scaler --> db

Known gaps and drift

  • Not live. No pool runner is registered. On 2026-11-03 a read-only GET /orgs/spadoople/actions/runner-groups returned HTTP 403 ("You must be an org admin or have the runners and runner groups fine-grained permission"). allows_public_repositories was not read and is not confirmed. The same token class already returns HTTP 403 on orgs/spadoople/actions/runners.
  • Docker memory. The role sets memory per host only. Docker pool runners inherit the docker guests' 6 GiB limit, so the four pool runners alone have a 24 GiB ceiling on a 16 GiB guest. During the overlap, ceilings are 84 and 80 GiB on the general guests (28 GiB maximum) and 78 GiB on each docker guest (16 GiB maximum). They were 52, 48, 54 and 54 GiB before the pool. These are ceilings, not reservations, but OOM counters need watching during cutover.
  • Apply ordering. Merging #591 does not register runners. A runner-role apply that selects an organization instance fails before the image build and before writing env when the org PAT is missing, empty, or identical to the per-repository PAT, and when the image already on the host is not r7 with an entrypoint that honors RUNNER_SCOPE. Building r7 in that same play does not satisfy the check. The r7 rollout is a separate owner-gated apply that selects no organization instance and restarts every selected runner, including homelab-lan-01.
  • Docker selector shape. Docker pool runners carry pool,docker and not ci, so DOCKER_RUNS_ON must be exactly ["self-hosted","Linux","X64","pool","docker"].
  • Exact pins to retire. RUNNER_FLEET_API_RUNS_ON pins homelab-ci-02 and Gmail pins its own runners by name. Both must move before phase 6.

What remains

Owner

  1. Gate A: confirm the Default group has allows_public_repositories: false. The 2026-11-03 runner-group GET returned HTTP 403, so this is not done.
  2. Gate B: create the org registration PAT and store it as github_runner_org_registration_pat in registration.sops.yaml. Do not reuse the per-repository PAT.
  3. Gate C: grant the github-actions-collector App (the scaler uses a copy of its key) organization Self-hosted runners: Read and accept the installation change. Do not change App settings from this repo.
  4. Gate D: approve the r7 rollout before any pool instance is applied. It restarts every runner, including homelab-lan-01. LAN goes last, and its exact-run guard must be re-verified. The role will not register an organization instance on a host still running r6.
  5. Decide the docker memory limit: keep 6 GiB, lower docker hosts to 4 GiB in phase 6, or fund a per-instance override in the role. The manifest does not change the existing 6 GiB docker host limit.
  6. Approve each cutover batch. #592 only prints the plan. It does not set CI_RUNS_ON or DOCKER_RUNS_ON, and the homelab batch does not touch RUNNER_FLEET_* or DEPLOY_RUNS_ON. Gmail's exact runner pins change only in that later batch.
  7. Approve phase 6 retirement separately. It is not done.

Agent (after owner gates)

  1. Run the r7 rollout and the pool apply in canary order, then verify guard hash, hook line and drop-in on homelab-lan-01.
  2. Run cutover batches: ci-toolkit and dnd_transcriber, then telegram_companion, then the rest, then homelab last.
  3. Validate each batch: queue wait, success rate, and no pool jobs on LAN, management or canary.
  4. Repeat the ADR-004 measurement two weeks after cutover and size phase 6.