Runner runtime tool-cache candidate — 2026-11-06

Summary

Repeated Python/Node downloads remain a measured full-CI bottleneck despite the completed native Action archive-cache rollout. The opt-in candidate prepares protected runtime seeds while retaining the existing writable, disposable job cache and standard setup Actions. Owner-approved preparation and native qualification passed in run 42280584653; activation, guest power changes and end-to-end speedup remain unperformed/unproven.

Evidence and state

Workstream	Verified state
Native Action archive cache	Previously qualified on all 24 pool runners; not replaced
Reddit kernel-start observation	Main run 42255798524: functional pass, timing failure at 372s gate / 373s run
Latest rebalance main	Run 42260102383: 155 modules / 3,215 ordinary cases plus full lanes passed; 330s gate / 331s run, warm kernels
Runtime tool cache	Actual-base build, offline native hits, isolated misses and protection/fresh-fixture checks passed; not selected or deployed
Scale-to-zero / deployment	Unchanged; no warm capacity, production deployment or forced guest transition

The latest main also contains unrelated NAS changes; its new test module receives the conservative duration fallback. Repeated reweighting is not a substitute for fixing runtime tool availability. Actual logs show Python/Node misses in every backend; one warm PR's Python download bracket alone took 41 seconds.

Candidate design

The runtime tool-cache runbook covers exact build/qualification boundaries. Reviewed release sizes/SHA-256 hashes were independently compared with publisher checksum files, API asset digests and real archives. The root seeder validates and extracts without executing upstream installers or upgrading pip; immutable seeds remain at their canonical paths. Runner-owned job-cache links select seeds without making the native miss root read-only. Writable additions remain container-local, never host-shared/promoted.

flowchart TD
    A[Reviewed current Action-cache image ID] --> B[Explicit legacy local candidate build]
    C[Reviewed runtime manifest and archive hashes] --> B
    B --> D[Protected canonical Python and Node seeds]
    D --> E[Disposable writable job-cache links]
    E --> F[Structural source and fixture tests]
    F --> G[Operator-approved actual-base build]
    G --> H[Offline native hits and isolated normal misses]
    H --> I[Separately approved canary and rollback]
    I --> J[Ordinary full CI qualification]

The local image-ID base contract deliberately uses DOCKER_BUILDKIT=0, matching the existing Action-cache preparation. BuildKit/Buildx bare-ID normalization is unsupported; no mutable-tag or registry workaround is introduced.

Owner-approved preparation route

The owner subsequently approved isolated actual-base preparation and native qualification, not activation or rollout. A task-PR-only workflow now targets same-repository PR #640, branch feat/runner-tool-cache and exact pool-docker-a-01; other routes remain inert. The fleet mutex is inside the fully gated job. The controller verifies the running base against its expected immutable identity, local Unix Docker socket and available disk before building from committed allowlisted files. It checks owned-container identity and cleanup, network-disabled native hits, isolated network-enabled different-version misses, and a fresh structural fixture without changing live selections or services. Run 42280584653 qualified source ec7d5015f1fea1cb48b8f80ee13efdf6b113b15c on that actual runner. Native Python 3.13.18 / Node 22.23.4 setup succeeded with networking disabled; a separate bridge-network fixture downloaded, installed and ran unseeded Python 3.13.17 / Node 22.23.3. Protected seed mutation attempts were denied, seed inventories remained unchanged, and a fresh offline container retained only seeded namespaces. The controller passed its ownership-checked cleanup gates; no independent subsequent daemon-wide absence query is claimed. The candidate remains untagged and unselected, so external GC can reclaim it; recheck before any later use. The active source, power and idle-policy exclusions above remain.

The first build exposed 0640 context files under the runner's 027 umask, making root-owned copied verifiers unreadable after USER runner. Explicit 0644 modes for committed nonsecret build inputs fixed it without weakening seed protection; regressions cover 022, 027 and 077. Failed attempts and the read-only stopped build inspection are retained in PR #640. The workflow now runs only on opening or source synchronization, not review-ready metadata transitions, avoiding a redundant candidate build when the qualified PR is marked ready.

Validation and remaining gates

Source tests cover size/digest/network bounds, hostile tar/link inputs, complete inventories, ownership/access contracts, safe output, synthetic dependency updates, and native-probe orchestration with fixtures. Both real upstream archives were prepared and structurally checked locally with synthetic ownership; that is not real Linux ownership or native Action execution. The local Docker daemon was unavailable and was not started; actual native qualification used the approved existing Docker runner instead. npm installation reported five existing audit findings (two low, three high); dependencies were not changed.

Homelab #637 tracks source review, actual-base build/native qualification and separately approved activation. Reddit #175 remains open for naturally cold/shared-load performance. Isolated preparation, offline native hits and normal unseeded fallback are now verified. The next gate is a separately reviewed image/selection/canary/rollback integration and explicit activation approval. Merging this source enables none of those operational steps automatically; active Dockerfile, Ansible/default image, entrypoint, guards and previous prepared records are unchanged.