Hyper-V runner owner actions¶

Owner follow-ups from the 2026-11-02 Hyper-V runner health pass. Every command below was run on CAPTAINKANGAPOO on 2026-11-02 unless the step says it needs an elevated session. Elevated steps were not run from the agent session because it is not elevated; their evaluation and validation commands were.

Canonical background: GitHub runner management, especially On-demand Hyper-V scaler and Ansible from CAPTAINKANGAPOO.

State at 2026-11-02 19:05 UTC¶

Item	State	Action
`gha-canary-a` MAC	Was `00155D29F013`, the same static MAC as `gha-docker-a`. Changed to `00155D29F016` while Off	Repair canary networking
`gha-canary-a` guest network	No ping or TCP 22 on the new MAC. Its netplan still matches `00:15:5d:29:f0:13`	Repair canary networking
`gha-docker-a`	Running, heartbeat OK, no ping or TCP 22. `tiktooker-docker-02` shows offline with `busy=true`	Recover gha-docker-a
Scaler runtime	`C:\ProgramData\homelab\hyperv-scaler\src` at `7fd53af`. Task `Running` as `renjamin`, S4U	Reinstall the scaler
PR #583	Merged as `ddc1527b`. The instance env template keeps the LAN job hook	Verify the LAN hook
Defender intelligence	Installed `1.463.93.0` at 02:42 local. KB2267602 `1.463.94.0` pending	Update Defender
`/opt/homelab` on infra-services	`ee2385c`, tracked edit only in `services/monitoring/compose.yml` (`grafana/alloy:v1.21.3`)	Fast-forward later
`gha-canary-a` memory	Live 2 GiB startup, 1 GiB minimum, 4 GiB maximum. Runbook table says 4/2/4	Decide canary memory

The duplicate MAC is the root cause of both SSH failures. Two running guests with one MAC on the GitHubRunnerNAT switch take each other's frames. The scaler log shows gha-canary-a running at 18:54 UTC while gha-docker-a had been up since 18:24 UTC; gha-docker-a dropped off the network in that window. The canary's SSH host key is unique, so the clone shared only the MAC and the netplan match.

Connections¶

Session checks¶

Elevation is required for the scaler installer and Defender install. Hyper-V cmdlets (Get-VM, Start-VM, Restart-VM, Set-VMNetworkAdapter) worked from a non-elevated session for renjamin.

([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
$PSVersionTable.PSVersion.ToString()

Expect True in an elevated window. PowerShell must be 7.x.

Guest SSH¶

Guests are reachable only from this host, through Windows OpenSSH and ~\.ssh\gha-runner-ssh.cfg. That config matches 172.29.240.* and forces user runneradmin, key github-runner-vm-admin, IdentitiesOnly yes, and IdentityAgent none. A plain ssh runneradmin@172.29.240.x uses the default config, which includes the 1Password agent, and fails non-interactively.

Guest	Address
`gha-general-a`	`172.29.240.11`
`gha-general-b`	`172.29.240.12`
`gha-docker-a`	`172.29.240.13`
`gha-docker-b`	`172.29.240.14`
`gha-mgmt`	`172.29.240.15`
`gha-canary-a`	`172.29.240.16`

$cfg = "$HOME\.ssh\gha-runner-ssh.cfg"
ssh -F $cfg -o BatchMode=yes -o ConnectTimeout=8 runneradmin@172.29.240.11 'hostname; systemctl is-active ssh; sudo -n true && echo sudo_ok'

Validated on gha-general-a: gha-general-a, active, sudo_ok. Confirm the resolved options without connecting:

ssh -G -F $cfg runneradmin@172.29.240.16 | Select-String '^(user|identityfile|identitiesonly|identityagent) '

Remote scripts piped from PowerShell must be LF-only. Here-strings are CRLF and produce $'\r': command not found:

$script = @'
hostname
uptime
'@ -replace "`r`n", "`n"
$script | ssh -F $cfg -o BatchMode=yes runneradmin@172.29.240.11 'bash -s'

Bounded port probe¶

Test-NetConnection hung on an unreachable guest. Use a timed TCP connect:

function Test-Tcp($ip, $port = 22, $ms = 3000) {
  $c = [Net.Sockets.TcpClient]::new()
  try { $c.ConnectAsync($ip, $port).Wait($ms) -and $c.Connected } catch { $false } finally { $c.Dispose() }
}
foreach ($ip in 11..16 | ForEach-Object { "172.29.240.$_" }) {
  "$ip ping=$(Test-Connection $ip -Count 1 -TimeoutSeconds 2 -Quiet) tcp22=$(Test-Tcp $ip)"
}

Hyper-V view¶

Get-VM gha-* | Select-Object Name, State, Uptime, Heartbeat | Format-Table -AutoSize
Get-VM gha-* | Get-VMNetworkAdapter | Select-Object VMName, MacAddress, DynamicMacAddressEnabled, SwitchName | Format-Table -AutoSize
Get-NetNeighbor -IPAddress 172.29.240.* | Where-Object LinkLayerAddress -ne '00-00-00-00-00-00' |
  Sort-Object IPAddress | Select-Object IPAddress, LinkLayerAddress, State

Every guest must have a distinct MAC 00155D29F0xx where xx is its last address octet. One MAC answering for two addresses in Get-NetNeighbor means a collision. IPAddresses on the adapter and Key-Value Pair Exchange stay empty or No Contact on healthy guests too, so they are not evidence of failure.

Runner state in GitHub¶

Busy and online state comes from each repository's runner list. The org list returns 403. Ephemeral docker-class runners can be NOT_LISTED between jobs.

$j = gh api repos/spadoople/tiktooker/actions/runners | ConvertFrom-Json
$j.runners | Select-Object name, status, busy

Inventory placements are in infra/ansible/inventory/github-runners.yml.

Hold a guest awake¶

The scaler stops any running guest whose last_needed is more than 900 seconds old on its next tick, including a guest you started by hand. On 2026-11-02 a manual Start-VM gha-canary-a was stopped about three minutes later. Refresh the timestamp for the guests you are working on:

function Set-ScalerLastNeeded([string[]]$Vm) {
  $path = 'C:\ProgramData\homelab\hyperv-scaler\state.json'
  $state = Get-Content $path -Raw | ConvertFrom-Json -AsHashtable
  $now = [DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds() / 1000.0
  foreach ($v in $Vm) { $state.last_needed[$v] = $now }
  $tmp = "$path.tmp"
  $state | ConvertTo-Json -Depth 3 | Set-Content -LiteralPath $tmp -Encoding utf8NoBOM
  Move-Item -LiteralPath $tmp -Destination $path -Force
}

Run it in its own PowerShell window and leave it there for the work window:

while ($true) { Set-ScalerLastNeeded gha-canary-a; Start-Sleep 45 }

Stop it with Ctrl+C when finished. The scaler then idle-stops the guest within 15 minutes. The scaler preserved the written value across its next tick in validation. Do not run Invoke-HyperVRunnerScaler.ps1 interactively, and do not disable the scheduled task to hold a guest awake.

1. Repair gha-canary-a networking¶

Evaluate¶

Get-VM gha-canary-a, gha-docker-a | Get-VMNetworkAdapter | Select-Object VMName, MacAddress

Expect gha-canary-a 00155D29F016 and gha-docker-a 00155D29F013. If the canary shows 00155D29F013 again, set it while the canary is Off:

Set-VMNetworkAdapter -VMName gha-canary-a -StaticMacAddress '00155D29F016'

Siblings configure eth0 in /etc/netplan/50-cloud-init.yaml with match: macaddress and set-name: eth0. The canary's copy still matches 00:15:5d:29:f0:13, so its NIC gets no address on the new MAC.

Remediate from the console¶

You need the runneradmin console password. If you do not have it, use the SSH fallback.

Start the hold-awake loop for gha-canary-a in its own window.
Start the guest and open its console:

Start-VM gha-canary-a
vmconnect.exe localhost gha-canary-a

Log in as runneradmin and fix the match:

ip -br link
sudo netplan get ethernets.eth0.match
sudo sed -i 's/00:15:5d:29:f0:13/00:15:5d:29:f0:16/I' /etc/netplan/50-cloud-init.yaml
sudo netplan get ethernets.eth0
sudo netplan generate && sudo netplan apply
ip -br addr show eth0
systemctl is-active ssh

netplan get must show macaddress: "00:15:5d:29:f0:16", address 172.29.240.16/24, and the 172.29.240.1 route. eth0 must be UP with 172.29.240.16/24.

SSH fallback without a console password¶

This uses the validated SSH path. It needs gha-docker-a Off so the old MAC is not shared.

Confirm gha-docker-a has no busy runners (see Runner state in GitHub), then Stop-VM gha-docker-a.
With the canary Off: Set-VMNetworkAdapter -VMName gha-canary-a -StaticMacAddress '00155D29F013'.
Start the hold-awake loop for gha-canary-a, then Start-VM gha-canary-a.
Connect with ssh -F $cfg runneradmin@172.29.240.16 and run the same sed, netplan get, and netplan generate commands as the console path. Skip netplan apply here. It would drop the session.
sudo poweroff, wait for State Off, then Set-VMNetworkAdapter -VMName gha-canary-a -StaticMacAddress '00155D29F016'.
Start-VM gha-canary-a and Start-VM gha-docker-a.

If step 4 is still refused while gha-docker-a is Off, the collision was not the only fault. Use the console path and check systemctl status ssh and ss -lptn 'sport = :22'.

Validate¶

"ping=$(Test-Connection 172.29.240.16 -Count 2 -TimeoutSeconds 2 -Quiet) tcp22=$(Test-Tcp 172.29.240.16)"
ssh -F $cfg -o BatchMode=yes runneradmin@172.29.240.16 'hostname; ip -br addr show eth0; sudo -n true && echo sudo_ok; cat /etc/machine-id'
Get-NetNeighbor -IPAddress 172.29.240.13, 172.29.240.16 | Where-Object LinkLayerAddress -ne '00-00-00-00-00-00'

Expect ping=True tcp22=True, hostname gha-canary-a, and .16 resolving to 00-15-5D-29-F0-16. Compare the machine ID with gha-docker-a; they must differ. With both guests running, .13 and .16 must stay reachable for 15 minutes.

Leave the hold-awake loop running and tell the agent. The agent patches the guest and rolls homelab-ephemeral-canary-01 to 2.338.2-r6 through the existing playbook path. Only after that, clear the README hourglass for guest creation.

Roll back¶

With the canary Off, Set-VMNetworkAdapter -VMName gha-canary-a -StaticMacAddress '00155D29F013' returns the previous state, including the collision. Prefer leaving the canary Off over running it on a shared MAC.

2. Recover gha-docker-a¶

Evaluate¶

Get-VM gha-docker-a | Select-Object State, Uptime, Heartbeat
"ping=$(Test-Connection 172.29.240.13 -Count 2 -TimeoutSeconds 2 -Quiet) tcp22=$(Test-Tcp 172.29.240.13)"

The docker-a runners are dnd-session-parser-docker-01, aiproject-docker-01, tiktooker-docker-02, telegram-companion-docker-01, and gmail-order-tracker-docker-01. Check them in their repositories. A runner that is offline with busy=true has a job that cannot finish while the guest has no network. GitHub fails it after the runner heartbeat timeout. Check the run in Actions before you restart if it matters to you.

Remediate¶

The canary must be Off or on its new MAC first, or the collision returns.

Restart-VM gha-docker-a -Force

The scaler log shows ACPI stop timing out for gha-docker-a on every idle stop on 2026-11-01 and 2026-11-02. The scaler then turns it off. That is a separate slow-shutdown finding. It does not block this restart.

Validate¶

Wait two to three minutes after the restart.

"ping=$(Test-Connection 172.29.240.13 -Count 2 -TimeoutSeconds 2 -Quiet) tcp22=$(Test-Tcp 172.29.240.13)"
ssh -F $cfg -o BatchMode=yes runneradmin@172.29.240.13 'hostname; sudo docker ps --format "{{.Names}} {{.Image}}" | grep gha-'

Expect five gha-* containers on local/github-actions-runner:2.338.2-r6, then the runners online in GitHub.

3. Reinstall the scaler from main¶

The installer pins the ProgramData runtime to the HEAD of the checkout you launch it from. Run it from main. Since 7fd53af, the code the scaler imports changed in scripts/runner_fleet/github.py (retry on a dropped HTTP read) and the 2.338.2-r6 tag in the adapters. Wake and idle-stop scripts are unchanged.

Evaluate¶

git -C C:\ProgramData\homelab\hyperv-scaler\src rev-parse HEAD
$t = Get-ScheduledTask -TaskName HomelabHyperVRunnerScaler; "$($t.State) $($t.Principal.UserId) $($t.Principal.LogonType)"
Test-Path C:\ProgramData\homelab\hyperv-scaler\config.ps1, C:\ProgramData\homelab\hyperv-scaler\fleet-read.pem

Validated before reinstall: 7fd53af76e2d64f8ed01990ed69e7ea266584001, Running renjamin S4U, both credential files True. The installer leaves an existing config.ps1 alone.

Remediate (elevated)¶

Open PowerShell 7 with Run as administrator and run the session check.
Put the repository on current main. Stop if git status shows changes you want to keep; do not stash or reset them for this.

cd C:\Users\renjamin\Documents\projects\homelab
git status -sb
git fetch origin main
git switch main
git pull --ff-only origin main

Run the installer:

.\infra\github-runners\hyperv-scaler\Install-HyperVRunnerScaler.ps1

It re-registers HomelabHyperVRunnerScaler, checks the runtime out at main, and sets AutomaticStartAction=Nothing on the six guests. It does not power guests off.

Validate¶

(Get-ScheduledTask -TaskName HomelabHyperVRunnerScaler).State
git -C C:\ProgramData\homelab\hyperv-scaler\src rev-parse HEAD
git rev-parse origin/main
Get-Content C:\ProgramData\homelab\hyperv-scaler\scaler.log -Tail 5
(Get-Item C:\ProgramData\homelab\hyperv-scaler\last-success.json).LastWriteTime

Expect Running, both SHAs equal, and last-success.json updated within the last minute. The installer may print a warning and fall back from S4U to Interactive. That fallback still runs hidden.

Roll back¶

From an elevated window on a checkout at 7fd53af, run the same installer.

4. Verify the LAN hook after role applies¶

PR #583, merged as ddc1527b, keeps ACTIONS_RUNNER_HOOK_JOB_STARTED=/etc/komodo-deploy/job-admission.sh in the homelab-lan-01 instance env when the github_runner role rewrites it. Without that line the exact-run verifier refuses to start the runner. Only run that role from a checkout that contains ddc1527b:

git merge-base --is-ancestor ddc1527b HEAD; "contains_583=$LASTEXITCODE"

Expect contains_583=0.

Validate¶

After any runner-role apply to infra-services, check read-only, without printing the env (it contains a registration PAT):

ssh -o BatchMode=yes infra-services-cursor 'sudo -n grep -c "^ACTIONS_RUNNER_HOOK_JOB_STARTED=/etc/komodo-deploy/job-admission.sh$" /etc/github-runner/instances/homelab-lan-01.env; sudo -n sha256sum /etc/komodo-deploy/job-admission.sh; systemctl is-active github-runner@homelab-lan-01'

Expect 1, hash a30f9450558ef734478141a6e9c459bc9af709a3b65de2f12e0e78275b62f632, and active.

5. Update Defender intelligence¶

Evaluate¶

Get-MpComputerStatus | Select-Object AntivirusSignatureVersion, AntivirusSignatureLastUpdated

KB2267602 is a rolling intelligence build that Defender also fetches on its own schedule. The non-elevated Windows Update COM install returned result code 4.

Remediate (elevated)¶

Update-MpSignature

Validate¶

Rerun the evaluation. The version must be equal to or newer than the pending build, and no reboot is requested. Do not install any update that requests a reboot of CAPTAINKANGAPOO as part of this runbook.

6. Fast-forward /opt/homelab later¶

This is not urgent. The running Alloy container already runs v1.21.3.

Evaluate¶

ssh -o BatchMode=yes infra-services-cursor 'cd /opt/homelab && git rev-parse --short HEAD && git status --short | head -n 20 && grep -n "image: grafana/alloy" services/monitoring/compose.yml'

Validated: ee2385c, one tracked edit in services/monitoring/compose.yml (line 199 grafana/alloy:v1.21.3), plus host-local untracked paths.

Remediate¶

Advance the checkout only under the host's normal pull and relay locks, per Komodo explicit deployment. main already carries grafana/alloy:v1.21.3, so the tracked edit matches upstream on that line. Preserve the untracked paths. Do not stash, reset, or clean. main also moves central cAdvisor to the ghcr v0.60.6 digest. That container was not recreated in this pass, so recreating it is a separate decision.

Validate¶

Run the evaluation again. Expect HEAD at the reviewed commit, no tracked change to the Alloy image line, and docker inspect alloy --format '{{.Config.Image}}' reporting grafana/alloy:v1.21.3.

7. Decide canary memory¶

Evaluate¶

Get-VM gha-* | Get-VMMemory | Select-Object VMName, DynamicMemoryEnabled,
  @{n='StartupGiB';e={$_.Startup/1GB}}, @{n='MinGiB';e={$_.Minimum/1GB}},
  @{n='MaxGiB';e={$_.Maximum/1GB}}, Buffer | Format-Table -AutoSize

All six guests use Hyper-V Dynamic Memory (balloon plus hot-add) with a 20% buffer. Only the canary differs from the placement table: live 2/1/4 GiB against documented 4/2/4 GiB.

Remediate¶

Either set the VM to match the table while it is Off, or update the table to the live values in a PR. No 14-day OOM evidence favors one; the single canary job is small.

Set-VMMemory gha-canary-a -DynamicMemoryEnabled $true -StartupBytes 4GB -MinimumBytes 2GB -MaximumBytes 4GB

Validate¶

Rerun the evaluation, then start the canary and confirm it boots and reaches SSH within the scaler's wake window.