Komodo image checks and reviewed pin updates¶

Status and boundaries¶

Merged in PR #506 as 6d6ff2e; bootstrap complete, optional credentials/execution validation and workload deployment pending. Current phase: edge_dispatch_blocked_github_queue.

Fleet API recovery overlay — 2026-10-27 10:03:15 UTC: the 08:14 LAN hold also blocked fleet dispatch validation through RUNNER_FLEET_API_RUNS_ON. Only that repo variable changed to ["self-hosted","Linux","X64","homelab","ci","homelab-ci-02"]; DEPLOY_RUNS_ON remains LAN, maintenance routing remains ["self-hosted","Linux","X64","runner-management"], and CI_RUNS_ON is unchanged. No host/VM, restart, label, permission or credential changes accompanied recovery. Health run 40485555074 succeeded (scope=homelab, dry_run=true, authorize_disruptive=false): Validate 17s and Observe 22s both actually ran on homelab-ci-02; mutation routes were skipped. 7 online / 2 offline / 9 discovered / 10 expected / 3 drifts is successful read-only collection, not a clean fleet. Fleet operations are usable again; do not blindly rerun unknown canceled mutations. The API selector also covers planning, scheduled checks and guarded watchdog; watchdog apply retains existing guarded approvals, not a blanket read-only contract. Preserve this temporary exact general-canary override until approved reassessment; it still depends on Hyper-V/scaler and one instance, not new always-on/redundant capacity. This does not unblock image deployment: edge remains queued with zero jobs, workflow disabled and normal Ansible timers active. The broad LAN hold is historical: service/container restored under exact-run quarantine at 10:49:47 UTC; now GitHub-online with non-deploying exact-LAN qualification successful and normal LAN job execution restored. No edge acceptance/deployment or global CI recovery is claimed. See recovery evidence and canonical routing runbook.

Current Komodo checkpoint — 2026-10-27: after #517 merged, the owner-approved deploy-reviewed-images-edge dispatch was accepted at 07:47:46 UTC as run 40469274207, attempt 1, head 35f9a67d218f78251cedb2dd26a1e1c9edfceee1. The workflow was briefly enabled then immediately disabled; a bounded 20.5-minute watch ended queued, zero jobs/check runs. A 60-second re-enable did not advance it; cancel/force-cancel returned HTTP 409 and in-place rerun HTTP 403. No retry was accepted or new dispatch made. Immediate disablement may have contributed; cause is unconfirmed, not evidence of server/runner failure. Edge execution is not accepted or a no-op: the job never started.

Historical broad hold — 08:14 UTC, superseded at 10:49:47 UTC: workflow was disabled_manually; owned drain marker /var/lib/github-runner/drain/homelab-lan-01 recorded 40469274207. The idle LAN runner was stopped after GitHub busy=false and no Runner.Worker (service failed/stopped, container absent, no job interrupted). Both Ansible pull timers were restored active and the owned /etc/homelab/MAINTENANCE removed. At that capture host HEAD remained 0f522f0, with no relay pending intent or edge RunProcedure (only old CreateProcedureUpdate), Sync at bootstrap 824c148; edge app/Core/Periphery container IDs/start times were unchanged. Prior qualification/bootstrap/grants remain complete: 51 Read + 5 Execute, 26 Procedures + 2 Actions.

Current exact-run quarantine — 2026-10-27 10:49:47 UTC: tested job-start hook and fail-closed pre-start verifier replaced the broad drain. Only the owned marker was removed after tests and backed up root-private. Actual LAN container runs as 1001:1001 plus group 10002, correct hook/read-only mount and no Docker socket; service active and GitHub verifies homelab-lan-01 online. Single non-deploying qualification run 40491873440 on main 54ca4da and exact LAN job 122148472245 both completed SUCCESS, verified via GitHub API, including Set up runner and Run Linux and macOS qualification checks. Normal LAN job execution is restored. komodo-deploy.yml stays disabled; ansible-pull-check.timer and ansible-pull-apply.timer active, MAINTENANCE, relay pending.json and drain absent. See the guard contract.

Reconcile the original run through inspection/GitHub support, not further cancel/rerun retries. Earlier browser access returned private-page 404 despite authenticated gh API access. Retain the guard after runner restoration; removal needs proof the orphan is terminal with no executable pending work plus deliberate reviewed follow-up. No automatic edge redispatch, workflow enablement or UI Procedure/Stack bypass. The owner merged docs-only recovery PR #519 as 4b2cb0c; rollout holds remain.

Current health limits (10:53:25 UTC): 49 running including the restored runner, 38 healthy, 11 without healthchecks, zero unhealthy/restarting; earlier DNS/TLS smoke good, not authenticated data acceptance. OneUptime and adapter October 27 B2 gaps are closed by explicitly owner-approved exact-snapshot copies; see the completed catch-up receipt. PR #520's local-retention fix is merged, but affected live units still lacked --retry-lock=5m at the 11:55 preflight. Normal convergence and next scheduled-run validation remain pending; no forced apply, restore/retention acceptance or broader mutation permission. Native image checking safely failed closed at 13/14 stacks / 32 services, not complete image coverage (see below). Gmail's absent Docker variable/unregistered proposed Docker runner is separate provisioning; the aiProject August 19 ghost was not caused by this work. Normal work may resume, without mass restarts or replay of unknown canceled work. See health limitations.

The manual GitHub check path passed on 2026-10-26 UTC for the edge cohort in run 40299854748: report only, publish skipped, no files changed. The credential correction is merged in #508, and the existing pair was migrated unchanged to /etc/komodo-deploy/komodo-deploy.env (1001:1001 0600, parent root:1001 0710, group search-only); the legacy file is retained. The prior approved cutover checkpoint remains migration/temporary runner-identity read-only preflight evidence, not actual recreated-runner acceptance. The owner merged PR #510 as 0416d8e, with all 26 PR-head checks successful; review and controlled runner activation are complete. Runner qualification reached runner_qualified_bootstrap_pending when credential, both-mode and natural-convergence acceptance passed at 2026-10-26 19:05 UTC, using the same actual container before, during and after the next natural apply, including root:docker 0750. The apply completed 19:04:57 UTC, exit 0, full recap ok=439 changed=29 skipped=98 failed=0. The separate operator-shell check-mode failure remains unresolved; this is not whole-host check-mode acceptance. At that 2026-10-26 acceptance capture, both timers were active, with no maintenance/drain markers or relay pending intent; deployment stayed disabled. No ResourceSync, grant change, deployment or Core/Periphery restart occurred in that window. See the historical acceptance record. The /etc/homelab:ro labctl mount and shared-parent role conflict remain separate and unchanged. This does not move the optional Core token. The extension is now bootstrapped: 26 live Procedures and 2 Actions, with 6 Repos, 21 Stacks and 1 Sync. No image-update dispatch token or publishing App credentials were provisioned in this work. Only services/komodo/image-updates.sops.yaml.example exists for the new credential; it is a schema example, not usable ciphertext. Preserve the separately completed deployment API preparation.

Historical bootstrap guard — 2026-10-26 20:24 UTC: PR #513 merged at 20:20:07 UTC as 645bc3c5467f82ef749048ef296b457a48476e39; all 26 checks passed. Native pending cache naturally refreshed to 645bc3c at 20:24 UTC (remote-content SHA-256 2551c36fe2b846ab5123e0c52f1a6823cc36f2c6ac3bbeaa2b84c0ca8e8b1c50): 18 creates + 13 updates, no deletes/deploy targets/errors/Variable/UserGroup changes. All four new retries (deploy-litellm, deploy-oneuptime-incident-adapter, deploy-paperless-stirling-bridge, deploy-firefly) had webhook_enabled=false in merged source and PENDING preview, not live-created Procedures at that capture; Sync was then unexecuted. The unsafe c9a0fd7 preview at 19:25 UTC is historical. Six legacy scoped webhooks are unchanged; Sync's webhook remains enabled, so the disabled workflow is not a global lock. At that capture, phase was runner_qualified_bootstrap_pending, runner online, workflow disabled; host was 8bf7aa5 and normal apply was left alone. Source-only six-profile scans and LiteLLM/Reddit checks passed, not DB-auth proof; preparation approval was then pending.

Historical bootstrap checkpoint — 2026-10-27 07:16:31 UTC: authorized bootstrap/grants completed. Reddit's fingerprint was refreshed with the same key, no env render/key change; all seven input --check calls passed. One external RunSync using existing homelab-deploy scoped Execute completed exact Update 6ae04f5aa10ff54a5801dc65 at 07:14:52 UTC, applying reviewed main 824c1487735a5ea0b8d25a1aafab8f0362d6fa92. Resource bytes were unchanged from #513; the reviewed preview was 18 creates + 13 updates, no deletes/deploys. Sync last_hash=824c148, pending resource changes/deploy targets 0, no errors. Existing admin browser/native UI API added exactly 15 Read + 3 Execute; the original 38 were unchanged. Bot readback verified 51 Read + 5 Execute = 56 and unchanged non-admin identity, with no broader permissions (see step 2).

All four new retry webhook flags are false in live resources; stored inline Actions exactly match reviewed source, with startup/schedules/webhooks disabled. Native check is now scheduled daily 03:00 America/Los_Angeles; legacy auto-update's schedule is disabled. This is configuration acceptance, not manual native/Action execution validation. No ProcedureRun/ActionRun, workload deployment, build, workload source pull or restart occurred; Core's normal Sync-cache fetch is not a workload pull. Host /opt/homelab remains 0f522f0; main #516's runner proposal is not applied there. Both pull timers are active, owned marker removed, no drain/pending relay intent. Core/Periphery retain 2026-10-25T11:33 starts; deployment workflow remains disabled_manually. See bootstrap and least-privilege grants complete.

Historical preparation — 2026-10-27 05:25:28 UTC: phase was five_inputs_prepared_reddit_baseline_pending, bootstrap_ready=false. The approved 05:24:58–05:25:28 UTC native root render/private HMAC/check preparation passed (exit 0) for incident-adapter, bridge, Firefly, Orders and PR Dashboard. Six rendered outputs including Orders OAuth JSON were byte-identical; five 0700 state directories/0600 keys and fingerprints had valid 1001:1001 ownership. All five scoped SOPS/config/running-env credential gates, actual Orders OAuth mounts in both containers and fresh password-authenticated read-only PG15 adapter/PG18 Firefly sessions passed. No rotation, SQL writes, HBA reload or network token validation occurred.

LiteLLM check passed; Reddit was stale before preparation and still failed at that capture. Both protected profiles' env/key/fingerprint bytes and metadata stayed unchanged. Read-only HMAC RCA exactly matched Reddit's clean current source 3f0b0c9 plus pre-#514 Compose: the gallery-worker read-only library change, not credential drift, made the baseline stale. Same-key fingerprint-only reconciliation had not yet occurred. Host advanced independently 8bf7aa5 → 0f522f0, preserving #514/#515; the pinned helper refused unexpected HEAD before fetch/merge. No operator checkout write/rollback; five-profile code/configs matched 645bc3c..0f522f0, and four dirty tracked files survived. Application/Core/Periphery IDs/start times stayed unchanged; healthcheck targets were healthy. Both timers were restored active and the owned marker removed; runner never drained/restarted, no relay pending intent. During that preparation window, workflow stayed disabled; no Sync/grants/deploy/Core change occurred. See the historical preparation record for root-private evidence; the separately authorized bootstrap subsequently completed above.

Normal weekly Dependabot configuration is unchanged. This is an on-demand complement, not a replacement, major-version resolver, merge bot or deployment bot. No Stack enables auto_update; automatic image deployment remains disabled. Checks/proposals never pull image layers, merge, deploy or force-redeploy workloads.

Path	Entry point	Effect
Native same-tag check	`check-image-updates` Procedure → same-named Action	Daily 03:00 `America/Los_Angeles`; native metadata checks for 14 fixed Stacks only
Registry version check	Manual `check-image-versions` → `request-image-pin-updates` Action → workflow `check`	Bounded public-registry tag/digest report; no source writes
Pin proposal	Manual `propose-image-pin-updates` → dispatch Action → workflow `propose`	Confirmed intent; fresh checks and dedicated App create a draft PR or report no changes
Deployment after human review/merge	Existing result-aware deployment workflow, chosen `deploy-reviewed-images-*` target	External Sync, then explicit homelab pull and conditional cohort reconciliation

The legacy Global Auto Update Procedure is retained for manual use with schedule_enabled = false and skip_auto_update = true. Its stored 03:00 schedule is disabled; the new native-check Procedure owns that daily slot. Backup Core Database at 01:00 and Rotate Server Keys at 06:00 remain scheduled. Neither Action has startup, schedule or webhook execution enabled; registry checks/proposals are manual-only. Manual GitHub check needs neither optional credential; the Komodo dispatch path still requires the optional Core PAT, and publishing requires the separate App.

What each check can establish¶

Native check: same tag, not newer releases¶

The fixed 14-Stack list is traefik, ara, homepage, monitoring, rexit, oneuptime-incident-adapter, authentik-outpost, adguard, litellm, paperless, firefly, stirling-pdf, phoenix, and sonarqube. This differs from the general 14-Stack deployment batch; do not substitute one allowlist for the other.

The Action calls CheckStackForUpdate with skip_auto_update = true, accounts for services against refreshed desired/deployed inventories, and reports safe per-service results. It does not pull/deploy, discover newer tags or edit pins. Digest-pinned images and local builds are explicitly unsupported_digest_pinned and unsupported_local_build. Missing, invalid or incomplete accounting fails the Action. Native update_available = false means not detected or unknown, never proof that an image is current. Even complete accounting has version coverage incomplete by design. Private Rexit remains on this native check list; safe unknown metadata is allowed, not a claim of registry access or permission to write Rexit versions.

2026-10-27 observed result: the image check safely failed closed at 13/14 stacks, 32 services. Rexit's latest inventory contains exporter + studio, while its deployed_services contains only studio; native checking prefers deployed_services. This is a latest/deployed service-accounting mismatch, not an auto_pull issue. Do not flip pull policy, weaken completeness checks or deploy Rexit to make the report green. The partial report is not full Action/version coverage or deployment acceptance; reconcile the inventories under a separately reviewed follow-up.

Registry policy: 13 image fields, six stacks¶

The fixed policy is services/komodo/image-update-policy.json. Only these Compose service image fields are eligible:

Cohort	Stack	Compose services	Fields
`edge`	`traefik`	`traefik`	1
`edge`	`homepage`	`homepage`	1
`edge`	`adguard`	`adguard`, `unbound`	2
`documents`	`paperless`	`broker`, `db`, `webserver`, `paperless-gpt`	4
`documents`	`stirling-pdf`	`stirling-pdf`	1
`finance`	`firefly`	`app`, `db`, `cron`, `importer`	4

Supported version tags select only higher same-minor patches, preserving the prefix and suffix exactly. No major/minor jump, flavor change or prerelease resolver.
Major-only aliases retain the same tag and refresh its digest only. This has no semantic patch guarantee: the publisher of that alias can move it across minor releases. Review release/migration implications even when the tag is unchanged.
Proposed references are image:tag@sha256:…. The checker hashes returned OCI index/manifest bytes, verifies registry digest metadata and existing immutable pins, and retains the top-level index digest rather than choosing one platform. It does not pull image config/layers or establish the running container version.
Errors, private registries, local-build services, digest-only references, unsupported tags/manifests and truncated pagination fail closed. Any failed/unsupported selected target prevents the entire proposal; no partial pin write or false “up-to-date” result. A successful current/no-changes report means only the bounded policy matched desired references, not that all releases or vulnerabilities were checked.
Registry bounds are 20 pages × 1,000 tags, 4 MiB per response, 512 requests, 300 seconds total and a 10-second socket timeout. No retries, redirects, proxy use, private-address access or local Docker credentials. The fixed public registries receive metadata requests; rate limits and outages remain possible.
Rexit, OneUptime, Komodo and local-build/source dependency updates are not automatic pin-update targets. Their existing promotion/maintenance/source-review paths remain separate; Dependabot's existing scope is not changed by this exclusion.

Proposal write boundary¶

The publisher may change only allowlisted Compose image scalars, matching literal old references in the affected README's Quick Reference → Image/Images table row, and append a future proposal record to services/komodo/image-update-history.md. It never rewrites arbitrary prose, examples, rollback instructions or unrelated docs. An unmatched README reference is left unchanged with a warning requiring human review. The history starts with a header only; proposal entries are desired-pin evidence, not deployment or application-acceptance receipts.

The GitHub App creates a new request-specific branch and draft PR. It does not force-push, overwrite an existing branch, enable auto-merge or deploy. An existing matching open PR can be linked; an orphaned branch or changed remote main stops the operation. A publish failure may leave partial GitHub state: reconcile it before retrying.

Owner bootstrap, in order¶

Steps 1–2 are complete; do not repeat activation, input preparation, Sync or grants. Completion PR #517 is merged; the approved edge attempt is blocked before job creation. Resolve original run 40469274207 before any new deployment dispatch; the LAN service is restored under exact-run quarantine, GitHub-online with non-deploying qualification successful and normal LAN jobs restored. Retain the guard until terminal/no-executable-pending-work proof plus reviewed removal; no automatic redispatch or cancel/rerun retry. App/Core PAT remain unprovisioned; steps 4–6 are separate owner gates, not automatic follow-ups. This documentation edit performs no live operations. Any future resource bootstrap must use reviewed merged source, not an unmerged feature checkout. The existing provider and infra-services-01 Sync must already exist; if absent, stop at the verified bootstrap boundary. Never disable KOMODO_UI_WRITE_DISABLED, edit Mongo or substitute a direct deployment.

1. Review source, runners and inline Action code¶

Source/CI review, #510 activation and credential/both-mode/natural-convergence acceptance are complete. Preserve manual-only main dispatch and the deployment bootstrap gates. The three reviewed flags remain true only in infra-services host_vars, with role defaults false. Credential-only common and homelab-lan-01-only runner applies succeeded, excluding observability; main site.yml still excludes github_runner. During the historical 12:06 activation, the drain marker was retained through apply and released after a fresh queue check; registration completed without a restart or retry. See the actual-runner evidence, not just the earlier temporary preflight. Do not repeat activation or recapture/rotate the unchanged pair.

The completed scoped acceptance preceded the separately authorized external Sync and least-privilege grants, now complete. Actual default-identity/group, read-only directory-mount and current-credential checks passed across both naturally occurring shared-parent modes; no live chmod or forced apply was used. Keep deployment disabled; preserve unrelated host files and /opt/homelab/.git/komodo-deploy/ state. Runner acceptance does not authorize optional image-publisher App/Core PAT provisioning, Core maintenance or workload deployment. Consumer rollback keeps the new mount plus explicit KOMODO_DEPLOY_ENV_FILE; old-mount-only rollback restores the bug. Legacy cleanup requires separate approval. CI_RUNS_ON must select trusted isolated CI runners without LAN/host mounts; the image workflow does not belong on the deployment runner. Its check job has read-only repository permissions and can run without either new credential.

Action sources live in services/komodo/actions/*.ts. At development time, scripts/render-komodo-actions.py embeds the two bodies into resources.toml as inline file_contents. Review source and generated TOML together; check drift with:

python3 scripts/render-komodo-actions.py --check

Core executes only the reviewed inline bodies imported by ResourceSync. Never execute runner-writable Core file paths, add a dynamic Action loader, or mount the checkout into Core. Core has no code-checkout mount. Actions execute with Core's administrative authority, making the reviewed inline bodies a critical trust boundary.

2. Bootstrap resources and grant least privilege¶

Completed 2026-10-27; not a pending approval or an instruction to replay. Five-profile preparation, same-key Reddit reconciliation and all seven input checks passed before one reviewed external Sync. It used the existing bot's scoped Sync Execute, not new administrator execution authority. The existing admin browser/native UI API then added exactly 18 per-resource grants: 15 Read + 3 cohort Execute. The original 2026-10-26 baseline of 38 grants (36 Read + 2 Execute) was preserved unchanged.

Root bot effective readback at 07:16:31 UTC verified 56 scopes = 51 Read + 5 Execute over 26 Procedures, 6 Repos, 21 Stacks, 2 Actions and 1 Sync, unchanged non-admin identity, empty user-specific/base permissions, and no added admin or Server/Build creation rights. Execute replaces Read for its resource; it is not an additional grant.

For future additions, keep external bootstrap separate from the relay: all declared Procedures, Repos, Stacks and Actions must already exist and be readable. Review the native pending plan, exclude independent UI/release/retry/Compose/Ansible writers and require the exact successful terminal Sync Update. Preserve the six legacy scoped webhooks; the still-enabled Sync webhook means workflow disablement is not a global lock. Preserve host 0f522f0, #514/#515, local edits and relay state; no rollback to older source.

Execute: infra-services-01, deploy-infra, deploy-reviewed-images-edge, deploy-reviewed-images-documents, deploy-reviewed-images-finance.
Read: every other managed Procedure/Repo/Stack/Action. This includes both Actions and the check/propose Procedures; the deployment bot does not need Execute on them.
The three cohort Execute grants were added after reviewed bootstrap. Independently authorize human operators to execute check/propose entry points. These are not bot-required grants; Execute on the proposal Procedure permits a draft-PR request, not merely a report.
Recheck effective/inherited permissions and non-admin identity. Keep base permissions empty and do not add broad Write/Execute grants. Native v2.3.4 parent execution uses internal administrative identities; child denial does not sandbox a changed parent.

3. Verify the manual GitHub check before optional credentials¶

After the workflow is on main, open Actions → Komodo image updates → Run workflow. Select main, operation check, and edge, documents, finance or all. Generate a fresh canonical lowercase UUID locally:

python3 -c 'import uuid; print(uuid.uuid4())'

Enter that non-secret value as request_id; leave confirmation empty. Record the exact run name Komodo image updates / check / cohort / uuid, with the selected cohort and generated UUID substituted. Require the matching successful, non-skipped check job and review its summary/artifact for every selected field. This route needs no Core token and no publishing App. Do not interpret a failed or incomplete report as “no updates”. Check reports/artifacts are retained for 30 days.

4. Provision the publishing App only if proposals are wanted¶

The owner must create/install a dedicated GitHub App on spadoople/homelab only with repository Contents: write and Pull requests: write (plus mandatory metadata read). Do not add administration or deployment permissions. Store its client ID in repository variable IMAGE_UPDATES_APP_CLIENT_ID and private key in repository secret IMAGE_UPDATES_APP_PRIVATE_KEY through approved private credential handling. Neither is provisioned by the source change. Never paste secrets into chat, command arguments, screenshots, logs or tracked plaintext files.

The workflow mints a short-lived, repository-scoped App token for publishing. There is no GITHUB_TOKEN fallback: App-created PRs must trigger ordinary PR CI. The workflow's normal GITHUB_TOKEN stays Contents-read-only. App permission scopes are broader than the path allowlist; reviewed publisher code enforces that narrower write boundary. Missing App configuration fails proposal publishing explicitly.

5. Optionally enable Komodo-to-GitHub dispatch¶

Skip this step if manual GitHub checks/proposals are sufficient. The owner may create a finite-lived fine-grained PAT, restricted to spadoople/homelab, with repository Actions: write and mandatory Metadata: read only. It needs no Contents: write, Pull requests or administration grant. Actions-write authority is broader than a single workflow; the Action's fixed repository/workflow/ref/argument whitelist is a code boundary, not token-level workflow scoping. Record expiry/rotation ownership.

Using the existing SOPS creation rule and approved private editor, create encrypted services/komodo/image-updates.sops.yaml with exactly KOMODO_IMAGE_UPDATE_DISPATCH_TOKEN plus SOPS MAC/age metadata. Use the example for schema only; do not commit populated plaintext. This file does not exist yet. Review and deliver only ciphertext to the intended main checkout. Do not put this PAT into shared compose.env, Periphery, /etc/komodo-deploy/komodo-deploy.env, or runner secrets.

On infra-services, drain checkout writers and verify the reviewed scripts and ciphertext before executing them as root. Required: Python >=3.11 with PyYAML, SOPS, the approved host age identity, and an existing root-owned /etc/homelab without group/other write. The renderer creates only an absent komodo-core leaf; it rejects unsafe existing directories rather than repairing their ownership/mode. From the trusted host checkout:

sudo python3 /opt/homelab/scripts/render-komodo-image-update-env.py --check
sudo python3 /opt/homelab/scripts/render-komodo-image-update-env.py
sudo python3 /opt/homelab/scripts/render-komodo-image-update-env.py --validate-existing

Run these individually and inspect each exit status: 0 means unchanged/valid, 3 means changed (or would change with --check), 1 means failure. Do not chain them with && and mistake expected status 3 for failure. Success is silent; --validate-existing is read-only and needs neither SOPS nor the age identity. The verified CLI also accepts --age-key-file; source/output paths are fixed.

The output is absolute /etc/homelab/komodo-core/image-updates.env, root:root 0600, inside a root-owned 0700 parent. The directory blocks UID 1001 traversal even through the LAN runner's read-only /etc/homelab mount. Keep it outside the recursively runner-owned checkout. Never relax these modes or print the file for verification.

6. Load the optional Core environment in controlled maintenance¶

Compose 2.24+ is required for the optional env_file (required: false). A missing file does not prevent Core startup; dispatch Actions fail clearly and point to the manual workflow. Rendering or restarting an existing container does not load a new environment: controlled Core recreation is required.

Follow Updating Komodo itself, including the approved maintenance window, idle/drained callers, rollback plan and verified unchanged database credentials. Host-side candidate validation must run as root, not the LAN runner. For an approved Core-only change, from /opt/homelab/services/komodo, the existing maintenance commands with root authority are:

sudo docker compose --env-file compose.env config --quiet
sudo docker compose --env-file compose.env up -d --force-recreate --no-deps core

Do not run recreation unless validation and all maintenance gates pass. This is a self-managed control-plane operation, not an image-proposal deploy/force path. Do not recreate Mongo or Periphery merely to load the Core-only token. Verify Core/database health and Periphery connection privately, then use a manual registry check to verify dispatch. No environment dump or secret prefix belongs in evidence. Future token rotation also needs controlled recreation; never confuse it with deployment API rotation.

Routine check and proposal¶

After the optional Core dispatch PAT is provisioned and loaded (and the publishing App configured for proposals), an authorized operator may execute check-image-versions; its fixed arguments request check for all. propose-image-pin-updates is manual and its reviewed definition supplies operation = propose, cohort = all and exact confirmation = PROPOSE_IMAGE_PINS. The Procedure does not add a separate interactive confirmation prompt: executing it is the proposal authorization. For a specific cohort, use the manual GitHub workflow or the authorized dispatch Action with only operation, cohort and, for proposals, that exact confirmation. No arbitrary target/ref overrides.

For manual GitHub proposals, choose propose, the cohort, a fresh UUID and type PROPOSE_IMAGE_PINS exactly. Both check and publish jobs must succeed; the publisher performs fresh checks rather than trusting an earlier report. A successful no-change run need not create a PR. Review any draft's complete diff, release notes, immutable pins, README warnings and history record; require ordinary PR CI before human merge. No merge/deploy is implied by a green proposal workflow.

The Action generates its own UUID and dispatches once. It polls only the exact Komodo image updates / op / cohort / uuid run on main, never “latest run”, with bounded discovery and a 35-minute total deadline (two 15-minute jobs plus queue allowance). A 200/204 dispatch response is acceptance, not completion. A timeout, cancellation, ambiguous match, truncated discovery or transport failure leaves an unknown/unverified outcome, not proof of cancellation. Record the request UUID, exact run name and any run/branch/PR IDs; inspect GitHub before considering a retry. Do not blindly redispatch with a fresh UUID or force-update an orphaned branch.

Deploy reviewed pins through the result-aware workflow¶

Current deployment hold: original run 40469274207 must be confirmed terminal and reconciled before a separately reviewed new deployment dispatch. LAN restoration under exact-run quarantine is not deployment approval; retain the guard until terminal/no-executable-pending-work proof plus reviewed removal. No automatic redispatch or UI Procedure/Stack bypass (see current checkpoint above).

After human review, CI and merge, use the canonical manual deployment trigger, not a direct Komodo UI Procedure/Stack execution or host relay bypass. Select main, enter DEPLOY_HOMELAB, and choose one approved target:

Target	Ordered work after successful external Sync
`deploy-infra`	Existing general graph; unchanged default
`deploy-reviewed-images-edge`	`PullRepo homelab` → `BatchDeployStackIfChanged` for `traefik,homepage,adguard`
`deploy-reviewed-images-documents`	`PullRepo homelab` → `BatchDeployStackIfChanged` for `paperless,stirling-pdf`
`deploy-reviewed-images-finance`	`PullRepo homelab` → `BatchDeployStackIfChanged` for `firefly`

A cohort reconciles whole changed stacks from current main, including non-image changes. It is not an image-only deploy, a PR-SHA deployment, or a force retry. Review all relevant Compose/config/source/secret and migration diffs since the last accepted state, not just the image PR; recheck main before dispatch. Drain independent writers throughout, including release/UI/Ansible and direct Compose callers. The shared relay lock, exact Sync/Procedure Update checks and pending-state recovery apply to all four roots. Current GitHub settings do not enforce environment reviewers or branch protection.

Inspect nested outcomes and application/data health, not merely Compose success. If postdeploy fails after Compose advanced its baseline, an unchanged conditional retry may skip the stack; use the existing separately reviewed recovery path, never change the cohort into an unconditional deployment. Record deployment acceptance separately from proposal history.

Evidence and remaining acceptance¶

Earlier metadata-only public-registry smoke checks for Traefik, Homepage and Stirling did not pull images, write pins, publish a PR, bootstrap resources or verify all 13 fields. PR #506 integration/CI evidence is recorded in the journal; it is not validation evidence for the follow-up credential-path correction.

On 2026-10-26 UTC, manual edge check run 40299854748 succeeded on merged main with request UUID 3a3c855b-381b-4787-b625-3c6fb6ec22ab. It reported Traefik v3.7 → v3.7.16 and initial digest-pin proposals for unchanged Homepage v1.13.2, AdGuard v0.107.79 and Unbound 1.22.0. Publish was skipped; the report's changed-file list was empty. No pins, update PR, image layers, Sync or deployment were produced. This verifies that manual read-only GitHub path only, not native/Action dispatch, all cohorts, proposal publishing or deployment. Deployment run 40299611768 failed earlier at credential loading before host fast-forward or Komodo mutation.

Review, runner activation, scoped convergence acceptance, input preparation, same-key Reddit reconciliation, all seven input checks, external Sync and least-privilege grants are complete. Phase is edge_dispatch_blocked_github_queue. Native-check scheduling is live; the current check is safely fail-closed at 13/14 stacks / 32 services, not full native Action acceptance. Manual-dispatch acceptance is not established by this partial result. Optional image-publisher App/Core PAT remain unprovisioned; controlled Core maintenance, correlated Action/proposal validation and publishing remain separate gates. Manual GitHub check is available.

Next: reconcile original run 40469274207 through GitHub inspection/support, without further cancel/rerun retries or redispatch. First explicit-workflow deployment, application smoke checks and unchanged retry remain unaccepted; a job that never started is not a no-op. Keep deployment disabled_manually and retain exact-run quarantine after runner restoration. Guard removal requires terminal/no-executable-pending-work proof plus deliberate reviewed follow-up. GitHub registration and non-deploying qualification are successful; normal LAN jobs restored. Both pull timers active, MAINTENANCE, relay pending.json and drain absent. Do not repeat bootstrap or add scheduled general deployment. See the audit and bootstrap completion record.