2026-11-02 — Hyper-V runner health¶

TL;DR¶

Five reachable Hyper-V guests took the pending Ubuntu upgrades with no reboot required, then moved to runner image local/github-actions-runner:2.338.2-r6 and Alloy 1.21.3. homelab-lan-01 is on the same image and the exact-run guard hash is unchanged. gha-canary-a still answers ping and refuses the SSH banner, so it was not patched or rolled.

What shipped¶

PR #580 merged as 89b201670127a872022af3f267cdf3723ab2f6cb before the live roll. It pins listener 2.338.2 image r6, Alloy 1.21.3, node_exporter 1.12.1, cAdvisor v0.60.6, SOPS 3.13.3, and the current ci-toolkit workflow SHA.

Guest	Apt	Image and agents
gha-general-a	Pending 0, no reboot; journal vacuum freed 2.8 GiB	13 runners on r6; Alloy 1.21.3; SOPS 3.13.3; node_exporter 1.12.1
gha-general-b	Pending 0, no reboot; journal vacuum freed 2.7 GiB	12 runners on r6; Alloy 1.21.3
gha-docker-a	Pending 0 after one SSH-recovery restart; journal vacuum freed 476 MiB	5 runners on r6; Alloy 1.21.3
gha-docker-b	Pending 0, no reboot; journal vacuum freed 933 MiB	5 runners on r6; Alloy 1.21.3; SOPS 3.13.3
gha-mgmt	Pending 0 from the earlier pass; builder cache prune reclaimed 29.59 GiB (80% to 46%)	`homelab-mgmt-01` on r6; Alloy 1.21.3
gha-canary-a	Not patched	SSH banner exchange is refused. ICMP to 172.29.240.16 works. The guest-creation hourglass stays
homelab-lan-01	infra-services had no pending apt	Listener 2.338.2, image r6, online and idle. Guard SHA-256 `a30f9450558ef734478141a6e9c459bc9af709a3b65de2f12e0e78275b62f632`. Drop-in and hook line still present

Central monitoring Alloy was grafana/alloy:v1.21.0 in the running container and v1.21.2 in the host compose file at /opt/homelab ee2385c. Grafana and collector data were backed up as restic snapshot 2e4a795c in /mnt/infra-backups/homelab-restic/monitoring. The alloy service alone was recreated on grafana/alloy:v1.21.3 and became healthy. The host compose file now has that image line as a local edit on top of ee2385c.

RUNNER_FLEET_API_RUNS_ON was not moved off homelab-ci-02. Dynamic memory and the 900s idle stop were not changed. komodo-deploy.yml was already active and was not dispatched or toggled.

flowchart LR
  apt[Apt on five reachable guests]
  canary[general-a canary then the rest of that host]
  docker[docker-b then docker-a]
  rest[mgmt then LAN]
  alloy[Central Alloy 1.21.3]
  apt --> canary --> docker --> rest --> alloy

Tuning¶

Fourteen-day job failure counts are application CI results, not a runner-infra change, so startup RAM and the 900s idle stop stayed put. The journal vacuum and the mgmt build-cache prune are the disk changes. Guest telemetry DNS was already pinned to 192.168.6.17 in /etc/hosts.

The on-demand scaler stopped idle guests during the roll, including gha-docker-b while its image build was in progress. Refreshing last_needed in the scaler state file kept the maintenance guests inside the existing 900s window. That refresh has ended. At 13:08 UTC the scaler's heartbeat showed no needed guests, and all six VMs were Off, which is the normal idle result. The ProgramData scaler checkout is still 7fd53af. Install-HyperVRunnerScaler.ps1 needs an elevated session and was not run.

gha-docker-a lost SSH twice while the VM stayed Running (banner exchange closed, then a later timeout). A restart restored SSH long enough to patch and roll it. gha-canary-a has the same banner failure and was left Off.

Root cause, found at 19:00 UTC: gha-canary-a and gha-docker-a had the same static MAC, 00155D29F013, on the GitHubRunnerNAT switch. The canary was running at 18:54 UTC while gha-docker-a was up, and gha-docker-a lost its network in that window. The canary's MAC is now 00155D29F016. Its netplan still matches the old MAC, so it has no network until that file is fixed. gha-docker-a was still unreachable at 19:02 UTC and was not restarted, because tiktooker-docker-02 still showed a job assigned.

The only pending Windows update is the Defender intelligence update KB2267602, which does not request a reboot. Download and install both returned result code 4 from this session, so it is still pending.

What remains¶

Owner steps for everything below are in Hyper-V runner owner actions.

PR #583 merged as ddc1527b. It keeps Alloy upgrades from reusing a stale extract directory and writes the LAN job hook back into the instance env.
Fix the canary's netplan MAC match, then patch and roll it.
Restart gha-docker-a and confirm its five runners return.
Re-run Install-HyperVRunnerScaler.ps1 from current main in an elevated PowerShell window so the ProgramData checkout is not left on 7fd53af.
Fast-forward /opt/homelab when that checkout is next advanced, and keep the local Alloy image line. It currently matches the r6 pin's v1.21.3 but the rest of that compose file is still ee2385c.