Komodo deployment uplift — 2026-10-25

TL;DR

All high/medium audit findings have source implementations or explicit operating policies. The platform remains Komodo/file-on-host, with secret-safe input selection, exact execution verification and independent host chains. PR #506 is merged; production bootstrap is complete and workload rollout remains pending. The first deployment attempt exposed a parent-directory permission conflict. PR #508 is merged, and the owner-approved credential migration is complete. The existing encrypted API pair is unchanged. PR #510 merged as 0416d8e with all 26 checks passed, and the approved LAN runner activation is now applied. That activation used /opt/homelab at 0416d8e, preserving #509 and unrelated work; the later bootstrap preflight found 8bf7aa5 (#511), then an independent writer advanced the host to 0f522f0 before the input-preparation writes. Runner identity, read-only directory access and non-admin authentication passed. At 2026-10-27 10:49:47 UTC, exact-run quarantine safely replaced the historical broad LAN drain. The service and actual container are running with the guard retained; GitHub verifies homelab-lan-01 online and non-deploying qualification successful; normal LAN job execution is restored. The edge orphan is still queued with zero jobs. Both Ansible timers are active, with no maintenance, drain or relay pending intent. Actual-runner credential/convergence acceptance is complete, including both naturally occurring shared-parent modes and a full subsequent apply. All five missing input profiles are now prepared and verified. Phase: edge_dispatch_blocked_github_queue. Reddit's desired fingerprint was reconciled using its existing key, all seven input checks passed, and the approved external Sync and least-privilege grants completed. Deployment stays disabled. No workload deployment or Core/Periphery recreation was performed by this rollout. Fleet API operations were subsequently recovered on a general runner while the separate edge run remains quarantined. The owner merged the docs-only recovery PR #519 as 4b2cb0c, without source/service behavior changes. The local-retention source fix is merged in #520; live convergence and next scheduled-run validation remain pending. The separately owner-approved exact-snapshot B2 catch-up is complete, closing only the two offsite snapshot gaps, not restore/retention acceptance. PR #518's bridge maintenance hold and all rollout restrictions remain.

Exact-run quarantine replaced the broad LAN hold

2026-10-27 10:49:47 UTC: installed and tested job admission for orphan edge run 40469274207, then removed only the owned /var/lib/github-runner/drain/homelab-lan-01 marker, backed up root-private. The broad drain/offline containment below is historical, not the current runner state. komodo-deploy.yml stays disabled_manually; the orphan remains queued with zero jobs. No redispatch, cancel/rerun retry or deployment occurred in this restoration.

• Guard /etc/komodo-deploy/job-admission.sh: root:1001 0750 under the existing root:1001 0710 directory, using its existing read-only bind mount. SHA-256: a30f9450558ef734478141a6e9c459bc9af709a3b65de2f12e0e78275b62f632.
• Instance environment: root:root 0600, with the unquoted literal entry ACTIONS_RUNNER_HOOK_JOB_STARTED=/etc/komodo-deploy/job-admission.sh. Its original is privately backed up and contains a PAT; never print either environment copy.
• Verifier /etc/komodo-deploy/verify-job-admission.py: root:root 0600, called by /etc/systemd/system/github-runner@homelab-lan-01.service.d/90-orphan-run-guard.conf using ExecStartPre=/usr/bin/python3 -I -B /etc/komodo-deploy/verify-job-admission.py. It fails closed on changed hash, modes, mount/launcher contract or lost hook config. Explicit runner-role apply may rewrite the environment; normal Ansible site does not apply that role. Preserve/revalidate the guard rather than bypassing the verifier.
• Real cached runner-image tests, network none, default 1001:1001, passed: orphan exit 1, another numeric homelab run exit 0, missing identity exit 2. Invalid/non-homelab identity is rejected too. Verifier positive and changed-hash rejection tests passed before release of the owned drain marker.
• Actual LAN container is running as 1001:1001 plus group 10002, with correct hook, read-only mount and no Docker socket; service is active and GitHub verifies homelab-lan-01 online.
• Single non-deploying qualification run 40491873440, main 54ca4da: validation placed on homelab-ci-02, matrix on homelab-mgmt-01, and exact homelab-lan-01 job 122148472245 completed SUCCESS, as did the run, verified via GitHub API. Set up runner (pre-job hook), checkout and Run Linux and macOS qualification checks succeeded. Normal LAN job execution is restored; no new test run was needed.
• Both ansible-pull-check.timer and ansible-pull-apply.timer are active; /etc/homelab/MAINTENANCE, relay pending.json and the owned drain marker are absent. Private cutover-state.json and edge-window-20261027.json record guarded resume and qualification run 40491873440; main phase remains edge_dispatch_blocked_github_queue. Temporary exact-homelab-ci-02 API routing, management/deploy selectors and watchdog's guarded mutation contract are unchanged.

Do not remove the guard because the runner is restored. Removal requires proof that the orphan is terminal with no executable pending work, plus deliberate reviewed follow-up. No automatic edge redispatch, cancel/rerun retry, workflow enablement or UI deployment bypass. The canonical guard contract preserves the edge quarantine despite successful registration and qualification.

Current health and limitations: 2026-10-27

The 10:53:25 UTC live snapshot found 49 running containers including the restored runner: 38 healthy, 11 without healthchecks, zero unhealthy or restarting. Earlier DNS/TLS smoke checks were good. This is runtime/route evidence, not authenticated application or data acceptance, and does not accept the blocked edge deployment.

• OneUptime and adapter October 27 B2 gaps are closed by explicitly owner-approved exact-snapshot copies; see the completed catch-up receipt. The local-retention fix is merged in #520, but affected live units still lacked --retry-lock=5m at the preflight. Normal convergence and next scheduled-run validation remain pending; this is not restore/retention acceptance or broader mutation permission.
• The image check safely failed closed at 13/14 stacks, 32 services. Rexit's latest service set is exporter + studio, versus deployed studio; native checking prefers deployed_services. This is an inventory/accounting mismatch, not an auto_pull issue; do not weaken accounting, flip pull policy or deploy merely to make the report green.
• Gmail's Docker variable is absent and the proposed Docker runner is unregistered: separate provisioning work, not caused by this recovery. Python/Maintenance job successes do not establish Docker capacity. Separate Gmail maintenance run 40489590509 placed validation and preview successfully, but management-runner apply failed SSH with No route to host for 172.29.240.13 / gha-docker-a, while inspecting the drain marker for gmail-order-tracker-docker-01. Operation drain, changed=0: no mutation executed. An offline/unreachable on-demand host is plausible; sleeping is not an established cause. No blind replay or waking unrelated VMs.
• The historical aiProject August 19 ghost predates and was not caused by this work.

The user may resume normal work within these limits. No mass restarts, automatic edge redispatch or replay of unknown canceled work; use intended scope and dry-run for any separately reviewed recovery. Guard removal and deployment remain separate reviewed follow-ups; successful LAN qualification does not authorize deployment.

Backup verification: 2026-10-27

Historical missing-copy evidence: read-only verification completed 11:29–11:30 UTC; all times below are UTC. Both main backups succeeded. All B2 snapshot metadata was scanned: 28 OneUptime / 12 adapter snapshots. Neither October 27 snapshot was found by originating ID or matching tree/time/path/host; both offsite copies were verified missing at that capture. The later approved catch-up closed both gaps. This establishes snapshot metadata presence/absence, not restore integrity.

Backup	Main snapshot	Saved October 27	Latest B2 snapshot at 11:29–11:30
OneUptime	4751b022	09:00:43	October 26 09:00:32
Incident adapter	5dd10199	09:02:55	October 26 09:01:45

Both failures were local restic forget --prune, before B2 copy, exit 11: OneUptime at 09:00:45, shared lock d9ceb188, PID 3763461; adapter at 09:02:56, shared lock 447634a2, PID 3772442. Fresh shared-lock contention is confirmed; exact process identity is unknown. Metrics collection overlaps the first failure and takes read locks: a plausible, not proven, cause.

At 11:30:13, both old PIDs were gone and local lock directories were empty; the active metrics reader was left untouched. No unlock, prune, copy, backup, restart or reset was performed during verification.

At that verification, --retry-lock=5m on local retention was proposed only; no mitigation or catch-up copies were applied. Subsequent source preparation on fix/restic-retention-lock-retry added the five-minute lock-acquisition retry to the shared template's first, local-retention post-step; it ran no copy, prune, backup, live apply or restart. Catch-up copies then required separate explicit authorization, subsequently granted and completed only for these two snapshots as recorded below.

B2 catch-up complete: 2026-10-27

Explicitly owner-approved exact-snapshot catch-up succeeded, completed 11:55:41.471 UTC. Preflight at 11:55:03 UTC confirmed both copies were still missing. Only the two fully resolved source snapshot IDs below were copied, individually, into their existing B2 repositories with normal locking and --no-cache --retry-lock=1m on the copy commands, not as a live unit alteration.

Backup	Exact source snapshot ID	Verified B2 destination snapshot ID	Verified UTC
OneUptime	4751b022d9d0ab59d875d440a52d74b752178c92cd784738b52d418b9d3fcb57	9fb98052adba53c9a5828e5e789179cabe58e7125421795a0e5a5670b11b0006	11:55:29
Incident adapter	5dd10199596eaeb8f47589cb8705432412c698d72f1710a8dd2fb6bb1a26ca3d	288782529f85afa345dea1e6ffea3b4f415051539d2a1f9324cdfa56d2fc8b02	11:55:41

Both destinations matched the exact originating snapshot ID (original) and corroborating tree, time, paths and host. The root-owned, mode 0600 private result receipt is /etc/homelab/komodo-deploy-bootstrap-20261026/b2-catch-up-result-20261027.json, with the plan alongside. This closes only these two offsite snapshot gaps: it does not establish restore integrity, successful retention or future scheduled-run acceptance. Original failed-unit evidence was deliberately left untouched. Post-copy systemctl inspection confirmed both backup units remain failed (Result=exit-code, ExecMainStatus=0); both ansible-pull-check.timer and ansible-pull-apply.timer remain active. No unlock, prune, backup, reset, restart or Ansible apply was performed in this catch-up. The scoped approval is completed, not general permission for further mutations.

PR #520 is confirmed merged as ea4acc863e4d2b3f963b8f4a34c2f90a66809a30. Its shared-template local-retention --retry-lock=5m bounds lock acquisition, not total retention runtime, for all managed jobs once normal Ansible convergence renders the units. At the 11:55:03 preflight, the affected live units still lacked that flag. Merged source is not live convergence or next scheduled-run acceptance; both remain pending. See the canonical source-fix contract. Normal timers/convergence remain on; no forced apply. Edge quarantine, disabled deployment and #518's bridge hold are unchanged; Gmail and Rexit remain separate exceptions.

Fleet API routing recovered with edge run quarantined

The owner's report that work was no longer placing exposed collateral from the LAN containment: fleet dispatch validation also used homelab-lan-01. Run 40472345328 queued at 2026-10-27 08:16 UTC on LAN/deploy labels after the 08:14 hold, then was cancelled at 09:31 with no runner/steps and all downstream routes skipped. The hold blocked that entry point; it did not stop the general, Docker or management fleet. Earlier fleet operations applied only a Gmail CI resume (run 40469102836) and previewed a Gmail Docker drain (40471032755, dry-run only).

Independent evidence distinguished placement from workload failures: homelab's 09:15 security workflow placed jobs on CI runners at 09:17, and Gmail's subsequent Python/maintenance jobs completed on its CI runner. Idle Hyper-V guests normally sleep after 900 seconds, so an all-offline snapshot alone is not a fleet outage. GitHub's public status reported operational at inspection; the individual edge orphan still remains unresolved.

At 10:03:15 UTC, changed only repository variable RUNNER_FLEET_API_RUNS_ON:

• Before: ["self-hosted","Linux","X64","homelab","lan","deploy"]
• After: ["self-hosted","Linux","X64","homelab","ci","homelab-ci-02"]

This uses an existing exact general instance, avoiding Docker/LAN routing. It covers API-only validation, observation/planning, scheduled checks and guarded watchdog operations; it does not mean every possible operation is read-only. Existing watchdog authorization/environment guards remain unchanged. DEPLOY_RUNS_ON stayed LAN-only, and RUNNER_FLEET_MAINTENANCE_RUNS_ON stayed on runner-management. No host/VM restart, label, credential, permission or maintenance apply was performed.

Verification was one read-only homelab health dispatch, run 40485555074, dry_run=true, authorize_disruptive=false, on main 54ca4da. It completed successfully at 10:04:43 UTC: validation took 17 seconds and observation 22 seconds, both actually on homelab-ci-02; other routes were skipped. Its report showed 7 online / 2 offline, 9 discovered / 10 expected and 3 drifts. This proves fleet API routing works again, not that every health finding or unrelated CI failure is fixed. No broad test campaign or fleet-wide restart was used.

The previous selector and proof run are recorded in local private artifact artifacts/cutover-20261026/fleet-api-routing-recovery-20261027.json. The recovery selector is temporary and single-instance, dependent on Hyper-V wake/registration; it does not establish always-on redundancy. At that capture, removing it would have restored the held-LAN fallback. Keep it until deliberate routing reassessment; the later guarded LAN restoration does not automatically revert this override. See the canonical recovery procedure.

At the 10:03 UTC recovery capture, original edge run 40469274207 had no jobs, workflow was disabled, LAN runner drained/offline, and normal Ansible timers active. The 10:49:47 UTC exact-run guard above supersedes that broad hold, not the API routing override. Fleet operations can be requested again, but an unknown cancelled mutation is not replayed automatically and no new edge dispatch is authorized. Current phase remains edge_dispatch_blocked_github_queue; the authorized docs-only recovery PR does not release the orphan or authorize deployment.

Edge dispatch blocked before job creation

After merging #517, the owner authorized only the edge cohort through the existing workflow. Main 35f9a67d218f78251cedb2dd26a1e1c9edfceee1 had all 27 PR checks passed. Compared with the host's 0f522f0, edge definitions/pins and deployment code were unchanged. The exact live Procedure was PullRepo homelab followed by conditional traefik,homepage,adguard; all managed resources were idle.

The normal Ansible apply finished before an owned window paused the two pull timers. The workflow was briefly enabled and one confirmed main dispatch accepted at 2026-10-27 07:47:46 UTC as run 40469274207, attempt 1, target deploy-reviewed-images-edge. Intake was immediately disabled again. The bounded 1,230-second watch expired with the run still queued, zero jobs and zero check runs, while the eligible runner was online/not busy.

Re-enabling the workflow and a bounded job-creation check did not advance it. Normal cancellation and force-cancellation both returned HTTP 409, saying Cannot cancel a workflow run that has not been queued yet. An in-place rerun request was refused with HTTP 403, This workflow is already running. No cancellation or retry was accepted; no second dispatch was issued. Immediately closing intake may have contributed, but the cause is not established. Do not claim an edge deployment failure, success or conditional no-op when no job started.

Verified: host HEAD remained 0f522f0, no relay pending intent existed, no edge RunProcedure Update was created (its only Update was the earlier CreateProcedure), and the last Sync remained the successful bootstrap at 824c148. Traefik, Homepage, AdGuard, Unbound and Core/Periphery container IDs/start times were unchanged.

Historical broad hold — 08:14 UTC, superseded at 10:49:47 UTC: to prevent a latent queued run starting outside the window, the idle LAN runner was stopped after checking GitHub busy=false and no Runner.Worker. Its owned /var/lib/github-runner/drain/homelab-lan-01 marker records run 40469274207; no job was interrupted. The service was stopped/failed with no container, and GitHub reported offline. Workflow intake was disabled and remains so. Both Ansible timers were restored active and only the owned /etc/homelab/MAINTENANCE marker was removed; routine convergence is not stranded while awaiting GitHub recovery.

Private edge-window-20261027.json and cutover state record edge_dispatch_blocked_github_queue. At that capture the instruction was to confirm terminal state and reconcile before retry or runner resume; the later exact-run guard permits guarded runner restoration without accepting the orphan. The connected browser could not view this private GitHub page, although authenticated REST access worked. The then-proposed UI inspection/cancellation was unverified. Current guidance is inspection/support reconciliation without further cancel/rerun retries, workflow enablement or direct Procedure execution. Keep the guard until terminal/no-executable-pending-work proof and deliberate reviewed removal; do not redispatch automatically.

The qualified runner configuration, completed bootstrap, 26 Procedures / 2 Actions and 51 Read + 5 Execute grants remain intact. Only the edge execution/acceptance is blocked. This checkpoint is included in the authorized docs-only recovery PR, without production source changes; record an accepted run only after verified recovery.

Bootstrap and least-privilege grants complete

The owner approved the coordinated bootstrap package and asked for focused checks rather than another broad test campaign. Completed on 2026-10-27:

• Rechecked the reviewed pending source at 824c148, byte-identical to the approved resource SHA-256 2551c36fe2b846ab5123e0c52f1a6823cc36f2c6ac3bbeaa2b84c0ca8e8b1c50. Scope remained 18 creations + 13 updates, with no deletes, deployment targets, variable/user-group changes or errors. Existing resources/Sync were idle.
• Opened an owned maintenance window and paused only pull timers. Held pull and relay locks. Refreshed only Reddit's desired fingerprint, using the existing key and current reviewed #514 inputs. Its key and rendered environment were byte/metadata-unchanged; no secret render or key rotation occurred. All seven native input --check calls passed.
• Issued one explicit external RunSync using the existing scoped deployment credential—not the general workflow or a deployment Procedure. The persistent execution journal recorded the exact ID before polling. Update 6ae04f5aa10ff54a5801dc65 completed Complete, success=true; operation acceptance and post-Sync verification ran 07:14:17–07:14:52 UTC. last_sync_hash=824c148, with zero pending resource changes/deployments.
• Through the existing administrator browser session and supported native permission API, added exactly 18 per-resource grants: 15 Read + 3 Execute. The original 38 grants were unchanged. Independent service-user readback at 07:16:31 UTC verified all 56 effective grants = 51 Read + 5 Execute. homelab-deploy remains enabled, non-admin/non-super-admin, with empty base/specific permissions and no server/build creation rights. No session credential was exported or new key created.
• Live inventory is now 26 Procedures, 2 Actions, 6 Repos, 21 Stacks and the existing Sync. Execute remains only on infra-services-01, deploy-infra, and deploy-reviewed-images-{edge,documents,finance}. Every other managed resource is Read-only to this identity; no Action Execute or downstream Write was added.
• Verified stored inline Action bodies equal reviewed source, and both Actions have startup/schedule/webhook execution disabled. The four new operator-only retry webhooks are explicitly disabled in live configuration. Existing scoped webhook contracts are unchanged.

The following schedules are now adopted in America/Los_Angeles:

Procedure	Live schedule
Backup Core Database	Enabled, daily 01:00
Global Auto Update	Schedule disabled; retained manual check-only definition
check-image-updates	Enabled, daily 03:00 native check-only Action
Rotate Server Keys	Enabled, daily 06:00

No Procedure/Action test run, workload build/pull/deployment or container restart was requested. The Sync's normal Core-side source fetch is not a host PullRepo or on_pull invocation. Core/Periphery retain their original 2026-10-25 11:33 UTC start times. Host /opt/homelab remains at 0f522f0; #516's runner proposal was not provisioned by this work. Restored both pull timers to active and removed only the owned maintenance marker; no drain marker or relay pending intent remains. The deployment workflow is still disabled.

Private receipts: bootstrap-window-20261027.json, bootstrap-input-acceptance-20261027.json, bootstrap-sync-result-20261027.json, and bootstrap-grants-acceptance-20261027.json under the existing root-private bootstrap directory. The phase is bootstrapped_deployment_pending. Validation was limited to the actual input/Sync/permission/configuration results and targeted documentation checks; no broad local regression suite was rerun for this step.

Next is owner review of the operational-record PR and a separately approved first explicit deployment with application acceptance, or optional image-publisher/App and Core-dispatch-token provisioning. The native image-check schedule is configured, not yet execution-qualified by a manual Action run. Optional credentials remain unprovisioned, and manual GitHub registry checks remain available independently.

PR #517 CI wording correction

The Python job in run 40466772147 found one documentation-contract failure: the bootstrap record changed the required phrase base permissions are empty to base permissions remain empty. Restored that explicit wording without weakening the test or changing the permission policy. All six focused provisioning/recovery contract tests passed; no broad local suite or production operation was repeated for this correction.

Five-profile input preparation complete

This historical input-preparation checkpoint precedes the completed bootstrap above. The owner approved the input-only window and asked for larger progress. The five scoped profiles were prepared and verified on 2026-10-27, 05:24:58–05:25:28 UTC without building, pulling, restarting workloads, executing Sync or changing grants.

Scope and checkout preservation

• Allowed the active normal Ansible apply to finish without interruption. Fresh native API reads found the existing managed baseline and Sync idle, with no nonterminal Updates; GitHub deployment remained disabled and no queued/running repository jobs were present at the preflight.
• Newer main included #514 Reddit gallery-worker library access and #515 Phoenix routing. The initial checkout plan pinned reviewed 645bc3c to avoid applying unrelated live-watched configuration. Its exact-HEAD guard then detected the host had independently advanced from 8bf7aa5 to 0f522f0 and refused before fetch/merge. No rollback or host checkout write was performed by this attempt.
• Confirmed the five profiles' rendering code/configuration, shared input engine and resource file were unchanged between approved 645bc3c and current 0f522f0. Used the current checkout rather than overwriting the newer work. Held both pull and relay locks for preparation; all four unrelated tracked edits stayed identical.
• Paused only the two pull timers with an owned maintenance marker. The runner was not drained/restarted. Both timers were restored active and the owned marker removed in the controller's cleanup; no relay pending intent was created.

Credential and preparation results

A root-private operator checker compared SOPS-derived candidates, resolved canonical Compose configuration and actual running environments in memory, with sanitized results only. All five passed. Orders' OAuth JSON also matched the canonical file and both running containers' pinned read-only file mounts. Firefly PostgreSQL 18 and incident-adapter PostgreSQL 15 passed fresh password-authenticated, read-only TCP sessions, role/expiry checks and first-matching password-HBA checks. HBA files were stable and predated postmaster start; no reload, SQL write or credential rotation was performed. External tokens were compared, not independently validity-tested.

The initial operator checker needed compatibility corrections for the existing root:docker 0440 age key, valid block-style SOPS recipient formatting and PostgreSQL inet::text CIDR output. Only the ignored operator artifact was corrected; host key permissions and encrypted inputs were not changed. Earlier checker failures were not reported as application failures or successful acceptance.

Profile	Native render + check	Rendered outputs	New private state
oneuptime-incident-adapter	Passed, check exit 0	compose.env byte-identical	Valid
paperless-stirling-bridge	Passed, check exit 0	.env byte-identical	Valid
firefly	Passed, check exit 0	.env byte-identical	Valid
orders	Passed, check exit 0	compose.env and OAuth JSON byte-identical	Valid
gh-pr-dashboard	Passed, check exit 0	compose.env byte-identical	Valid

Each new .deploy-inputs directory is 1001:1001 0700, with key/fingerprint 1001:1001 0600. The native serial path used no Docker/nsenter fallback; Docker was used only for read-only compatibility inspection and existing-container probes. Private backups of the six pre-existing rendered files remain under the bootstrap staging directory. Subsequent credential checks passed again. Target applications remained running/healthy, and their container IDs/start times were unchanged. Core/Periphery were not recreated.

Existing Reddit baseline remains explicitly failed

LiteLLM's read-only check passed. Reddit's check now reported missing or stale fingerprint, despite both private state files being present and its source checkout clean. The overall input audit remained nonzero; it was not relabelled as a global pass. The five independent target profiles passed their own credential/DB gates and were prepared within the authorized write scope. At that checkpoint Reddit's failure still blocked bootstrap; no Reddit/LiteLLM refresh was performed then. Their environment, key and fingerprint bytes and metadata were unchanged.

A read-only HMAC comparison isolated the cause: the stored Reddit fingerprint exactly matches current application HEAD 3f0b0c9fd6b520fc8c12e3f73b1e0b93aaf42e46 plus the pre-#514 Compose bytes. Keeping all other input components unchanged reproduces the stored value. Thus the stale baseline is explained by the reviewed gallery-worker read-only /library Compose change, not by a secret mismatch or missing key. No key, MAC or input contents were printed, and no state was written by that diagnostic. A refreshed desired fingerprint would still not prove deployment/application success.

The phase then was five_inputs_prepared_reddit_baseline_pending, with bootstrap_ready=false. The proposed next approval bundled: inspect/reconcile Reddit's desired fingerprint using its existing key, without environment render or key rotation; verify all inputs; re-review fresh Sync source and the 31-change preview; execute the external Sync and verify its exact successful Update; then add only 15 Read + 3 Execute grants (51 Read + 5 Execute total). That bundle was subsequently approved and completed as recorded above. Workflow enablement, workload deployment and optional publisher/Core credentials remain excluded.

Private receipts under /etc/homelab/komodo-deploy-bootstrap-20261026/ include input-preparation-window-645bc3c.json, input-preparation-result-20261027.json, per-profile logs, before/after credential-check reports and input-output-backups-20261027/. Retain these alongside prior cutover evidence. All documentation/agent-guidance surfaces are updated locally for the operational record; no new source/configuration commit or PR was created in this window.

Retry guard merged: input preparation ready for approval

This historical readiness checkpoint precedes the completed five-profile window above. The owner merged PR #513 at 2026-10-26 20:20:07 UTC as 645bc3c5467f82ef749048ef296b457a48476e39; all 26 PR-head checks passed. The local reviewed checkout is at that merge. Resource validation and inline Action agreement passed again.

The first native read still showed cached c9a0fd7. Without forcing refresh, Sync's normal poll advanced its pending preview to 645bc3c at 20:24 UTC. The single resource file exactly matched merged source (SHA-256 2551c36fe2b846ab5123e0c52f1a6823cc36f2c6ac3bbeaa2b84c0ca8e8b1c50). All four operator-only retry definitions explicitly had webhook_enabled=false. The preview still contained 18 creations and 13 updates, no deletions/deployment targets, errors, variables or user-group changes; managed/delete remained false. This closes the source-guard merge/re-preview gate, not live resource bootstrap: none of the new Procedures or Actions was imported. Private evidence is /etc/homelab/komodo-deploy-bootstrap-20261026/bootstrap-preview-645bc3c.json.

Read-only preparation checks

• Native root Inputs.snapshot(rendered=False) scans passed for all six new profiles. They read source/config/ciphertext but did not decrypt, render, publish fingerprints or invoke a renderer. The Orders and PR Dashboard scans also passed their clean-source requirements, including ignored-file exclusion.
• Existing LiteLLM and Reddit fingerprint --check calls both returned 0. Preserve their keys and outputs rather than unnecessarily refreshing them. Fingerprint checks are not proof of live database authentication or credential expiry; Reddit's separate fingerprint also does not establish SOPS/rendered-env equality. Those secret/runtime prerequisites remain distinct.
• Native Python was 3.12.3, SOPS was available, and someone resolved to 1001:1001. Every service directory was 1001:1001, mode 0755 or 0750. Five profiles had no state directory/key/fingerprint. Existing output files were single-link regular files, with no symlink ancestors; renderer-owned data/client directories already had the expected metadata. Recheck in the approved window.
• Host /opt/homelab remained 8bf7aa5 and normal Ansible apply was active; it was not interrupted. No checkout update, timer pause, drain/maintenance marker, environment write, fingerprint refresh, Sync, grant change or restart occurred.

Proposed input-only approval scope

Approval had not yet been given at this checkpoint. The proposed sequence was to wait for active convergence to finish and coordinate conflicting writers. Safely advance the host to reviewed merged source under the appropriate locks, preserving unrelated edits and #511. Before output writes, recheck destination safety, source stability and private credential consistency with the actual running consumers; require fresh read-only database authentication where applicable. Stop on mismatch, expiry or failed authentication; this approval must not imply credential rotation, database changes or repair.

Prepare only the five missing profiles, serially via native host Python with a clean environment—no Docker/nsenter fallback. Each refresh is followed by --check:

Profile	Approved candidate renderer outputs and metadata
oneuptime-incident-adapter	Service compose.env, root:root 0600 under root execution
paperless-stirling-bridge	Service .env, root:root 0600 under root execution
firefly	Service .env, root:root 0600 under root execution; existing native SOPS backend
orders	Service compose.env, 1001:1001 0600; /etc/homelab/orders/google_client.json, 10001:10001 0400
gh-pr-dashboard	Service compose.env, 1001:1001 0600

Each may create a service-owned 0700 .deploy-inputs directory, private 0600 key and fingerprint, and private temporary render/state files. Existing valid HMAC keys must be reused; missing-key/unsafe-state failures do not authorize resetting state. Orders/dashboard renderers also ensure their existing client/data directory modes/owners: /etc/homelab/orders root:root 0755, Orders/dashboard /var/lib/homelab/* directories 10001:10001 0750, and the dashboard gh child 10001:10001 0700. This is not recursive data ownership repair.

Recheck LiteLLM and Reddit read-only; no refresh of their existing outputs/keys is proposed. Stop at the first error and inspect partial effects: a refresh can replace outputs even when fingerprint publication fails or reports unchanged. It is not a transaction or an application-health receipt. No builds, source/image pulls, workload/container restarts, Sync, grants or workflow enablement are included. Input preparation, then bootstrap/grants, then deployment remain separately gated. No owner credential recapture is needed for this proposed scope. Documentation and agent-guidance updates are retained locally for the next operational record.

Bootstrap preflight and operator-only retry guard

The owner merged PR #512 at 2026-10-26 19:17:51 UTC as c9a0fd7b199a97bcac25418d3df71d9600753fd7, then asked for the next step. This historical preflight preceded #513's merge above and inspected the bootstrap proposal and permissions; it did not execute Sync, grant permissions, prepare inputs or deploy anything. Runner qualification remains complete; phase remains runner_qualified_bootstrap_pending.

Reviewed live preview

The 19:25 UTC native GetResourceSync response for infra-services-01 (6a0407af4c98985723c7d07b) reported pending revision c9a0fd7. Its single remote file was services/komodo/resources.toml, byte-identical to reviewed merged source (SHA-256 a1b2d9971648556eed7e2de92d42b158e782d1596da4bf4501801506aa115be5). Full resource validation and inline Action agreement passed.

Preview operation	Exact scope
Create 16 Procedures	Six host/application children; four operator-only retries; six image check/proposal/cohort Procedures
Create 2 Actions	check-image-updates, request-image-pin-updates
Update 6 Stacks	orders, gh-pr-dashboard, oneuptime-incident-adapter, litellm, firefly, paperless-stirling-bridge
Update 3 Repos	homelab, orders-source, gh-pr-dashboard-source input-refresh hooks
Update 4 Procedures	deploy-infra graph/webhook, plus adoption of Backup Core Database, Global Auto Update, Rotate Server Keys

Total: 31 changes = 18 creations + 13 updates. There were no deletions, pending deployment targets, errors, variable changes or user-group changes. managed=false, delete=false, source path/linked Repo matched expectations, and Sync was idle. The existing Sync webhook remains enabled; the disabled GitHub deployment workflow is not a global lock on independent callers.

The two Action data.proposed preview strings are not portable standard TOML: Python tomllib rejects their generated multiline JavaScript escaping. This is a preview/export compatibility limitation, not a source parse failure. Upstream v2.3.4 RunSync refetches and parses source into typed resources, then passes typed Action configuration to create/update handlers; it does not reparse those cached preview strings. Do not rewrite Action regexes to repair the display. Conversely, a preview is not immutable authorization: recheck fresh source and pending changes immediately before any approved execution.

Stop condition: new operator-only retry webhooks

The four new force-retry Procedures omitted webhook_enabled; in Komodo v2.3.4 omission defaults true. That would add webhook-triggerable unconditional DeployStack paths despite the source calling them operator-only:

• deploy-litellm
• deploy-oneuptime-incident-adapter
• deploy-paperless-stirling-bridge
• deploy-firefly

The owner authorized publishing the correction on fix/komodo-operator-retry-webhooks. PR #513 added explicit webhook_enabled = false on exactly these four; it did not execute a live Sync. The existing retry validator rejects omission or true, with mutation tests for both defects and preservation of the six legacy scoped release/retry/canary webhook contracts. The initial 93 focused tests passed, followed by 275 combined graph, relay, activation and documentation tests, full resource validation and inline Action agreement. Markdown, strict MkDocs/source-built checks (220 pages), all 11 CI generator checks plus the hook-only remediation-target check, changed-file pre-commit hooks and git diff --check also passed without generated drift. No stages, schedules, Actions or legacy webhook policies changed. At that checkpoint the correction was unmerged; the subsequent merge/re-preview is recorded above. It still has not been imported into live resources. A one-off UI edit is not a durable substitute.

Input and permission preparation

Read-only file-presence checks found five missing fingerprints: incident adapter, Paperless–Stirling bridge, Firefly, Orders and PR Dashboard. LiteLLM and Reddit fingerprints existed as private 1001:1001 0600 files, but existence does not prove freshness or validity. The host checkout was 8bf7aa5, not the earlier activation-time 0416d8e; preserve #511 and unrelated host work. No checkout, rendering, fingerprint refresh or host Repo pull was performed by this preflight. Prepare/verify required inputs only in the separately approved window before Sync adds the tracked files; cache refresh can otherwise retain missing-file errors.

Native ListPermissions plus GetPermission on all current grants confirmed 36 Read + 2 Execute, with no effective-level differences or specific grants. The enabled homelab-deploy identity remained non-admin/non-super-admin, with empty base permissions and server/build creation disabled. Desired post-bootstrap access is still 51 Read + 5 Execute: add 15 Read and 3 Execute grants, not broad Write/admin/Action Execute. The additional Execute roots are only deploy-reviewed-images-edge, deploy-reviewed-images-documents, and deploy-reviewed-images-finance; existing Execute stays on the Sync and deploy-infra.

At that checkpoint the next action was review/merge of the guard PR. That is now complete; obtain separate approval for input preparation and external bootstrap/grants, drain independent writers, then verify the exact Sync Update before adding only the reviewed grants. Sync can register schedules and refresh Stack caches; it is not an inert file import or a deployment-success receipt. The new daily native image check is check-only; no immediate workload deployment is requested by the reviewed source. Optional App/Core-token provisioning, control-plane recreation and workload deployment remain later gates.

Private redacted evidence is retained as /etc/homelab/komodo-deploy-bootstrap-20261026/bootstrap-preflight-c9a0fd7.json. Deployment stayed disabled, the runner stayed online, and no timers, markers, permissions, runtime configuration or control-plane containers were changed.

Natural convergence acceptance complete

On 2026-10-26, the owner authorized observation and a docs-only follow-up PR, not deployment. The 18:24 UTC inspection found eight successful normal apply recaps since activation, including 18:12:11–18:23:15 UTC. The actual runner's credential checks still passed, but historical apply logs could not establish a credential read specifically during the restrictive parent mode. A bounded, read-only observer therefore watched the next scheduled cycle without forcing it.

Observation (UTC)	Evidence
18:26:17, after the previous full apply	Actual runner passed at /etc/homelab = root:root 0755
18:54:43	Next timer-driven apply started naturally; no timer, marker or service changes by the observer
18:57:37–18:57:38	Parent naturally became root:docker 0750 (GID 987); actual runner passed credential/access/authentication checks in that state
19:01:44	Parent naturally returned to root:root 0755
19:04:57	Full apply completed, exit 0: ok=439, changed=29, skipped=98, failed=0, unreachable=0
19:05:00	Actual runner passed again after the completed apply at root:root 0755; observer finished within its one-hour bound

All three probes used the same actual container, cdd2e124a9d2…, started at 15:09:30 UTC, runner ID 25235. Each checked the default UID/GID 1001:1001, groups [1001,10002], no Docker group/socket, all capabilities dropped and no-new-privileges, and the same-path read-only /etc/komodo-deploy directory bind. Host SOPS/output and actual-container validation returned 0; credential read passed, directory listing was denied, and write-open returned EROFS without writing. HTTPS GET /user authenticated enabled, non-admin/non-super-admin homelab-deploy. The shared parent was stable before/after each probe, and container identity was unchanged. No credentials or raw API responses were exported.

This closes the previously outstanding both-mode and post-activation convergence gate. Private state now records runner_qualified_bootstrap_pending, with both_parent_modes_verified and post_activation_convergence_verified true. The root-private evidence is /etc/homelab/komodo-deploy-bootstrap-20261026/natural-convergence-20261026T182615Z.jsonl. The observer did not chmod the parent, force an apply, pause timers, create/remove maintenance or drain markers, restart/register the runner, rotate credentials or invoke an execute endpoint. Its only writes were private observation records; the subsequent state update records the result, not a deployment enablement.

Two separate operational facts must not be conflated with this acceptance:

• Since the initial activation snapshot, four runner listener exits returned 0, followed by systemd restart/reregistration and listening for jobs. The current runner/container IDs differ from the historical 24873/f469fae… pair; that is not evidence of credential-preflight failure. The observed cycle retained the same current container throughout.
• The 17:50:00–17:51:58 UTC check-mode invocation exited 2 at operator-shell : Set operator shell home, reporting an undefined value (ok=135, changed=3, skipped=76, failed=1). This unrelated failure was not fixed. The successful full-apply recaps and credential probes do not establish that every host check is clean. The shared-parent role conflict and labctl traversal also remain a separate unresolved issue; only Komodo's dependency was removed.

Deployment intake remains disabled, normal timers remain active, and maintenance, drain and relay-pending markers remain absent. Core/Periphery retain their original 2026-10-25 11:33 UTC start times. No ResourceSync, grant change, workload deploy, image pin update or optional Core/publisher credential provisioning was performed. Next is separately reviewed external bootstrap of the merged 26 Procedures / 2 Actions and least-privilege grants, then separately approved deployment/application acceptance. PR #512 published the activation and acceptance record; it does not change inventory, runtime scripts, secrets or deployment controls. The docs branch is based on later main 8bf7aa5 (#511), preserving its backup and LiteLLM changes. The acceptance-time host checkout remains 0416d8e; updating this docs branch did not advance or deploy the host.

Runner activated: remaining convergence gate

This historical activation checkpoint predates the completed convergence evidence above. The owner authorized waiting up to 30 minutes, then continuing the approved runner-only activation. The earlier apply finished normally at 2026-10-26 12:01:20 UTC, exit 0; no backup/restore work or Ansible process was interrupted. The check service's last invocation at 11:49 UTC also exited 0, but this alone does not distinguish a full check from a lock-related skip and is not all-checks-passed convergence evidence.

• Rechecked deployment disabled, runner offline/not busy, no queued/running/waiting repository workflows, and no pending relay intent. Read-only native API checks confirmed the existing 10 Procedures, 6 Repos, 21 Stacks, 0 Actions and Sync were idle with no nonterminal Updates. No execute endpoint was called.
• Recorded a fresh private activation window, created an owned maintenance marker and paused only the two pull timers. Held both pull and relay locks for the verified-bundle fast-forward from 9240794 to 0416d8e. There were no incoming local conflicts. All four unrelated tracked edits were byte/mode/owner-identical afterward, and the observed local-file set remained intact. No staging, reset, stash, cleanup or backup-artifact relocation was used.
• Effective combined inventory confirmed the three persistent flags true only on infra-services, with only homelab-lan-01 allocated. The narrow common credential apply reported ok=8, changed=0, skipped=2, failed=0; both renderer checks returned 0. The original credential and legacy copy were retained unchanged.
• The runner-only wrapper excluded github_runner_observability and selected only homelab-lan-01: ok=39, changed=3, skipped=6, failed=0. Changes were the installed launcher, runner Docker-GC helper file and starting the selected service. No image rebuild or GC operation occurred; the GC timer stayed disabled on LAN. The owned drain marker remained present, so the new launcher waited with no container/registration during configuration.
• After successful apply and fresh queued-work checks, released only the owned drain marker at 12:05:55 UTC, retaining its private archived content. No redundant service restart or adapter resume was issued. The 90-second GitHub registration wait expired; immediate follow-up found the runner online/idle, with zero service restarts and no launcher error signals. Registration was not retried. Runner ID: 24873; container started 12:06:26 UTC, image local/github-actions-runner:2.336.0-r5.
• Actual-container checks used its default UID/GID 1001:1001, groups [1001,10002], no host Docker GID 987 or Docker socket, all capabilities dropped and no-new-privileges. The same-path /etc/komodo-deploy directory bind was read-only. Host SOPS/output check and actual-container --validate-existing returned 0; credential read succeeded, directory listing was denied, and write-open returned EROFS without writing bytes. HTTPS GET /user authenticated as enabled homelab-deploy, non-admin and non-super-admin. No secrets were logged.
• The actual runner observed /etc/homelab as root:root 0755 only. Its new-path access under a naturally occurring root:docker 0750 state, and after subsequent normal convergence, remain unverified. Do not change the live parent permissions or rotate credentials to manufacture acceptance. #508's isolated lifecycle tests and the earlier temporary preflight remain separate evidence.
• Removed only this window's owned maintenance marker and restored both timers to their prior active states. Runner service remains active with zero restarts; GitHub reports online/not busy. Drain marker and relay pending intent are absent. Private phase is runner_activated_pending_convergence. At the 12:09 UTC inspection, the next normal apply was scheduled for 12:32:44 UTC; recheck the schedule before observing rather than forcing a broad apply.

Private records under /etc/homelab/komodo-deploy-bootstrap-20261026/ include activation-window-0416d8e.json, activation-checkout-preservation-0416d8e.json, the two phase logs/result receipts, released-drain-marker-0416d8e.json, and actual-runner-acceptance-0416d8e.json. Keep these and all legacy staging/credentials until separately approved cleanup.

Core/Periphery still have their original 2026-10-25 11:33 UTC start times and :2 containers. No ResourceSync, Procedure, permission grant, workload deployment, image pin update or optional image-publisher/Core credential provisioning occurred. At this checkpoint, deployment intake stayed disabled through remaining convergence observation and later bootstrap/grant/deployment gates. Handoff docs and agent guidance were prepared locally for PR #512. Post-activation validation passed: 94 focused activation/documentation tests, Markdown fix/check, strict MkDocs and source/built-site checks (220 pages), and git diff --check. These documentation checks do not close the live convergence gate.

Activation review complete — waiting for convergence

This historical waiting checkpoint precedes the successful activation above. PR #510 merged at 2026-10-26 11:27:39 UTC as 0416d8ead3e4cdfc78577a877172c2a3b09dbeaa. All 26 PR-head checks, including the final CI gate, completed successfully before merge. The owner's review/merge gate is complete; the remaining prerequisite for the already-approved runner window is a fresh idle/concurrency preflight, not another review of the same flags.

• A normal ansible-pull-apply.service run started at 11:22 UTC and remained activating/start after a bounded 180-second idle wait. The waiter returned 75 (still busy); the production apply was neither killed nor interrupted. No new maintenance marker was created or timers paused in this follow-up.
• /opt/homelab was rechecked at 9240794 with the four unrelated tracked edits and observed host-local files intact. Normal ansible-pull uses a separate /var/lib/ansible-pull/homelab checkout: its progress, or its consumption of the merged common settings, does not establish /opt/homelab advancement or runner activation. The ordinary site playbook does not apply github_runner.
• Workflow komodo-deploy.yml remains disabled. homelab-lan-01 remains offline, not busy, with no container and the owned drain marker present. No relay pending intent was present. Private state now records activation_reviewed_waiting_for_convergence and activation_applied: false.
• Prepared and verified a Git bundle containing only the reviewed 9240794 → 0416d8e advance; staged it under .git/komodo-cutover-20261026/. Staged the root-private activation-0416d8e.yml wrapper under the existing bootstrap directory. These are preparations, not a checkout update or Ansible apply.
• The wrapper selects 10 common credential tasks only with the existing komodo-deploy-credentials tag, then only github_runner with the explicit homelab-lan-01 allowlist. Unlike the stock runner playbook, it does not import github_runner_observability. Source path overrides account for its private location; all three activation flags still come from persistent inventory. Task listing and all five activation tests passed. No runner-role apply occurred.
• Read-only prerequisites passed: working Docker; expected checkout ownership; dedicated credential metadata unchanged; pinned cached runner image provenance; installed Dockerfile/entrypoint matching reviewed source; pinned SOPS checksum; required registration ciphertext, host identity and labctl file present. No credential values were printed. These checks do not establish registration or actual-runner acceptance.

Resume only after convergence finishes: repeat the idle/queued-work checks, acquire the appropriate locks, preserve host files while advancing to reviewed main, apply common credential provisioning, then apply only the LAN runner role with its owned drain marker retained. After successful apply and fresh registration preflight, release that marker deliberately. The waiting new launcher continues; do not add a redundant service restart or replay the earlier timed-out drain. Deployment remains disabled through actual-runner/convergence acceptance and the separate bootstrap/grant/deployment gates. At that checkpoint, the updated handoff was local pending this follow-up publication; merged activation settings were unchanged.

Approved cutover checkpoint — 2026-10-26 UTC

This historical checkpoint precedes the #510 merge recorded above. The owner explicitly approved cutover. Work proceeded only through credential migration and activation-PR preparation; the next boundary then was owner review/merge before runner recreation or registration. Bootstrap and workload deployment remain separate gates. The isolated clone is on ops/komodo-credential-activation; the original dirty checkout and unrelated work remain untouched.

• Recorded original control states privately in /etc/homelab/komodo-deploy-bootstrap-20261026/cutover-state.json, then disabled komodo-deploy.yml, paused Ansible pull timers and created an owned maintenance marker. No pending relay intent was present.
• Drained idle homelab-lan-01 through the native maintenance adapter, after checking GitHub busy=false and no Runner.Worker. Its 30-second stop call timed out; subsequent inspection confirmed the container was gone and the service stopped with exit 137. No job was interrupted. The adapter did not write a success replay receipt; do not blindly replay drain or interpret systemd's failed status as a running container. The drain marker remains at /var/lib/github-runner/drain/homelab-lan-01.
• Allowed the already-active Ansible apply, including restore-drill bootstrap work, to finish normally at 10:51 UTC (Result=success, exit 0). It was not interrupted. The check service's exit 2 at 05:48 UTC predates cutover and remains unresolved; restored timers are not evidence that every convergence check passed.
• Staged the renderer and ciphertext from immutable merged #508 under private merged-6ab7660/; verified their bytes also match newer main 9240794. Explicit root-only --migrate-legacy returned 3 (changed). Private comparison confirmed the new and retained legacy credentials match approved SOPS; no values were logged, rotated or recaptured. New directory: /etc/komodo-deploy, root:1001 0710; file: komodo-deploy.env, 1001:1001 0600.
• A temporary, automatically removed container from the configured cached local/github-actions-runner:2.336.0-r5 passed read-only preflight: UID/GID 1001:1001, supplementary 10002, no Docker GID 987, all capabilities dropped, no-new-privileges, same-path read-only directory mount. Credential read succeeded; directory listing was denied and write-open returned EROFS. HTTPS GET /user authenticated as enabled, non-admin homelab-deploy. This was not a registered or recreated runner and is not actual-runner acceptance.
• The generic safe-pull attempt timed out after 90 seconds before advancing HEAD. It was not retried with a longer timeout and its source was not modified. A tested targeted fast-forward instead held both Ansible-pull and relay locks and used a verified local Git bundle. Only dirty services/rexit/compose.yml overlapped incoming main; its bytes already matched incoming main. It was archived and only that identical path staged, without a worktree reset, before fast-forwarding from 034fe1d to 924079477549e3962a3f4b2c6c9073aeb7fdaf72. #509's LiteLLM credential-rendering changes were retained.
• Unrelated tracked edits remained byte/mode-identical: backups/restore-drills/rexit.py, backups/restore-test.sh, scripts/prox-mem-soak-finalize.sh, scripts/prox-mem-soak-sample.sh. Observed host-local files and backup artifacts remained in place. Private preservation records, the Rexit archive and checkout output remain under the bootstrap directory; Git bundles remain in .git/komodo-cutover-20261026/. Nothing was broadly reset, stashed, cleaned or removed.
• The actual merged /opt/homelab/scripts/render-komodo-deploy-env.py subsequently returned 0 for both --check and --validate-existing.
• Removed only this cutover's maintenance marker after verifying its content and ownership; restored both Ansible pull timers to their prior active state. Private state is now awaiting_activation_review. Fresh inspection confirmed the deployment workflow remains disabled_manually, the runner offline/not busy with no container, the drain marker present, and no relay pending intent. Routine convergence is not held while waiting for review.

This activation proposal enables all three flags only in persistent infra/ansible/inventory/host_vars/infra-services.yml: common_komodo_deploy_api_reviewed, common_render_komodo_deploy_env, and github_runner_komodo_deploy_api_enabled. Role defaults stay false. The only allocated runner on that host is homelab-lan-01. These source settings have not been applied; the host remains on pre-activation 9240794 at this checkpoint.

Core/Periphery retain their original 2026-10-25 11:33 UTC start times and existing :2 containers (observed version 2.3.4). No ResourceSync, Procedure, workload deploy, new grants, image pin update or optional publisher/Core credential was invoked or provisioned. Live bootstrap from 10 Procedures / 0 Actions to the merged 26 Procedures / 2 Actions remains pending. Keep legacy credentials and private staging/archives until separate cleanup approval.

At that checkpoint, next was review of the activation PR and CI, then coordinating the narrowly scoped common credential task and only homelab-lan-01 runner maintenance from reviewed merged inventory. See activation review and runner resume. Do not dispatch deployment or bootstrap merely because the migration passed.

Activation proposal validation

• All five activation tests passed locally, including effective Ansible inventory from both inventory files, strict host/group/inline declaration checks, disabled role defaults and the sole LAN-runner allocation. These are configuration tests, not an Ansible apply or live-runner acceptance.
• Local relay, operational-doc, inventory-generator, credential-activation and #509 literal-rendering regressions: 220 cases, 219 passed, one opt-in Compose skip. Initial documentation wording assertions were corrected without weakening tests.
• Disposable Linux Python 3.13.17/Bash 5.2.37, UID/GID 1001:1001: 576 cases, 567 passed, nine skipped, across Komodo (364), relay (88) and documentation (124). The snapshot excluded actual secrets and Git metadata; the container had no network, host credentials or Docker socket, with capabilities dropped and a read-only root. Temporary artifacts were removed. The nine skips cover four opt-in Docker lifecycle probes, three native-Ansible cases, one Compose case and one root-only UID boundary. Prior #508 lifecycle evidence remains separate.
• All four safe-pull tests passed on Linux. Their initial local failures were the macOS Bash 3 declare -A limitation, not evidence of production-helper success or a reason to modify its implementation. An initial Linux harness noexec tmpfs issue was corrected in the disposable environment only.
• All 11 CI generator checks plus the hook-only remediation-target check passed without generated drift. Inline Action agreement and Komodo resource validation passed (21 Stacks, 6 Repos, 26 Procedures; two inline Actions).
• Markdown fix/check, strict MkDocs and source/built documentation checks passed for 220 pages. Changed-file pre-commit hooks passed, including YAML/Ansible, owner-reference, runner-topology and Markdown checks; git diff --check passed. Activation PR CI and actual-runner acceptance remain separate gates.

Scope and packaging status

The owner approved all high/medium work in the capability audit, and later explicitly authorized an isolated branch, commit, push and pull request to package the change. That authorization is not a deployment or merge instruction.

The original working tree starts at 4e7a8d1 and contains both the Komodo proposals and unrelated work. Initial implementation compared selected runtime/test files with production fec3dbc720289c7371eeb6654b1f6f4dc9817241, preserving Reddit's existing optimization. Remote main subsequently advanced to 7e7f79b6 and includes newer operational/secret/recovery documentation plus manual-only dispatch policy.

PR #506 packaging used a fresh isolated clone at artifacts/komodo-deployment-pr, based on 7e7f79b6, on branch feature/komodo-deployment-uplift. The fresh clone avoids the original checkout's malformed origin/main 2 ref; the original working tree is untouched. Proposals were merged selectively, not copied over newer-main contracts. Implementation commit 1f7d483f485f2fcc858ab65ae48557de8c3cd620 is pushed and PR #506 was merged by the owner at 2026-10-26 09:26:10 UTC as 6d6ff2ef028abfabb31ea392673fbd5a33902916. All 26 merged-head checks passed. Integrated local validation is recorded below; the merge is not workload rollout acceptance. PLAN.md, unrelated documentation/history/navigation and workload placement are unchanged. The later image-update source extension is recorded separately below; it is not included in the earlier implementation commit or validation counts.

Area	Source contract	Live state
Input-aware deployment	Six additional private HMAC profiles, render-before-compare, predeploy verification, conditional selection	All seven input profiles verified; first deployment/application acceptance pending
API relay	Exact Update completion, bounded deadline, no blind retry, persistent lock/intent, full managed visibility and busy checks	#508 migration and #510 persistent activation complete; actual-runner credential/both-mode/natural-convergence acceptance passed; external bootstrap and 56-grant scope verified; first deployment acceptance pending
Isolation	Independent host chains and independent application children; Firefly before apps	Graph synced; first execution/failure-isolation acceptance pending
Versions	Matched Core/Periphery/helper 2.3.4 pins and pinned LXC installer recipe	All four versions observed 2.3.4; helper tag cached; Compose pin application pending
Bootstrap ownership	Earlier three bootstrap schedules declared; new extension replaces legacy image schedule with daily native check, still check-only	Adopted and verified through the approved external Sync; scheduled execution acceptance remains separate
Missed triggers	Explicit operator-reviewed manual workflow recovery	No autonomous full-deploy schedule introduced

Post-merge checkpoint — 2026-10-26 UTC

• Merge verified at 6d6ff2e; all 26 checks on the merged PR head passed.
• The host remained at 034fe1d during inspection, with unrelated tracked edits and host-local files. They were not reset, stashed, removed or moved.
• Live Komodo still contains 10 Procedures, 0 Actions and 1 Sync. The merged 26 Procedures and 2 Actions have not been bootstrapped. Core/Periphery remain on their existing :2 containers, unchanged since 2026-10-25 11:33 UTC.
• Manual deploy run 40299611768 failed at credential loading with Cannot read private Komodo API credential file. This was before host fast-forward, ResourceSync or Procedure execution; no relay pending intent was created. No automatic retry or permission workaround was applied.
• The legacy file /etc/homelab/komodo-deploy.env itself remained 1001:1001, 0600. At this dated capture the LAN container actually ran as 1001:1001 with supplemental group 10002, not host Docker group 987. It could not traverse /etc/homelab while that directory was root:docker, 0750. During the active Ansible apply, the directory changed to root:root, 0755, and the same container could read the file again. No credential rotation was needed.
• Source evidence: common/tasks/ansible-pull.yml and monitoring-deadman/tasks/main.yml manage the shared directory as root:root, 0755; operator-tools/tasks/labctl.yml manages it as root:docker, 0750. Stable credential access must be fixed declaratively and tested across convergence. Do not add the runner to Docker, make credentials broadly readable, or rely on a temporary directory chmod.
• Read-only image run 40299854748 succeeded on merged main for edge; request UUID 3a3c855b-381b-4787-b625-3c6fb6ec22ab. The publish job was skipped and the report's file list was empty. It reported Traefik v3.7 → v3.7.16, and initial digest-pin proposals for unchanged Homepage v1.13.2, AdGuard v0.107.79 and Unbound 1.22.0. No pins, update PR, image layers, Sync or deployment were produced.
• The image-update App and optional Core dispatch token remain unprovisioned. Runner-access correction and the controlled bootstrap are the next gates.

These are historical observations before the approved cutover checkpoint above. At this earlier checkpoint, only follow-up PR preparation was authorized.

Credential-access follow-up — 2026-10-26 UTC, source only

The #508 source-preparation work used the isolated artifacts/komodo-deployment-pr checkout on fix/komodo-runner-credential-access. The original checkout and unrelated work are untouched. The five already-dirty checkpoint docs are retained; old path/runtime observations above and below remain dated history, not rewritten as new-path evidence. This follow-up has separate validation evidence; PR #506's earlier counts do not validate the correction. Source preparation does not authorize production cutover.

Follow-up validation

• Full Linux Python 3.13.17 discovery: 2,056 passed, zero failures/errors; 37 cases skipped for opt-in or unavailable native tools (2,093 cases discovered). The run used an unprivileged, network-disabled disposable snapshot without host secrets, host mounts or a Docker socket. Missing ssh-keygen was supplied only during disposable image setup; no production dependency was changed.
• All 4 explicit Linux/Docker lifecycle probes passed, separately from the default-skipped suite: 19/19 new-path reads across reordered/repeated/partial old parent policies; the old path reproduced EACCES in all 11 restrictive-mode checks.
• 32 atomic replacements remained visible on subsequent opens with unchanged directory inode and no partial outputs. Eight injected precommit failures preserved previous state. The read-only directory-mounted runner saw replacements without restart; owner write/delete returned EROFS, directory listing/entry mutation were denied, and unrelated UID 10001 could not read the credential.
• The probes used only synthetic values, UID/GID 1001 plus group 10002 (no Docker group), zero effective capabilities and no network. UUID-labelled test containers and volume were removed after validation; no live host was involved.
• Focused local suites: 299 tests, no failures/errors, one Linux-only skip. Migration fault tests separately passed under Linux UID 1000 and UID 1001 so ownership-injection coverage does not depend on the CI runner's identity.
• Strict redacted Gitleaks staged-diff scan: zero findings, no new allowances. The original real encrypted API source and its key material were not changed.
• Markdown and strict MkDocs/source-built documentation checks passed (220 pages). Changed YAML/Ansible tasks passed lint; existing configuration/line-length notices remain warnings. git diff --check passed.
• Independent security review found no high-confidence code blocker. Review/merge, persistent opt-ins, actual-runner convergence acceptance and deployment remain separate owner-controlled steps.

PR #508 CI fixture correction

Both failing test jobs in run 40305132743 failed the same migration-fixture assertion. mkdir(mode=0755) was filtered by CI's restrictive umask to 0750, but the test expected unchanged 0755. Migration correctly rejected the unsafe directory; this was a fixture setup assumption, not a production permission repair or credential-access failure.

The fixture now explicitly sets its intended mode after private creation. It checks that migration leaves unsafe 0755, 0750 and 0770 directories untouched, and its mode helper is tested under umasks 0022, 0027 and 0077. All 20 migration tests passed under each mask; 73 focused tests passed under 0027 with one Linux-only skip. A disposable Linux Python 3.13.17 run as UID/GID 10001 with umask 0027 passed all 357 Komodo tests, with eight optional/native-tool skips. No production code, runner umask, host permissions or security checks were relaxed.

PR #508 merge and cutover readiness

PR #508 merged at 2026-10-26 10:37:20 UTC as 6ab7660333766c5a2e27bcba31c0030e2e163d40; all 26 PR checks passed. Read-only post-merge inspection found the host checkout still at 034fe1d, the legacy API file still 1001:1001 0600, and /etc/komodo-deploy absent. The LAN runner is online and idle, with no drain marker or relay pending intent. Core and Periphery retain their original start times. An Ansible apply is active, so cutover must coordinate with it rather than race a checkout update or stop it midway through a task.

Unrelated tracked edits and host-local files remain untouched. Of the observed tracked dirty paths, only Rexit Compose overlaps the incoming change set; its bytes matched incoming main at inspection. Recheck all conflicts at cutover; this is not permission to reset, stash or discard host work. The three persistent activation flags are not set on the LAN host; defaults remain false.

No migration, runner recreation, Sync, deployment, maintenance flag or timer change was performed in this readiness pass. The next owner gate is approval for the controlled credential/runner cutover and its persistent activation/documentation change. Workload deployment and control-plane recreation remain separate gates.

Source contract and cutover

The source correction removes Komodo's dependency on the shared /etc/homelab parent rather than changing its conflicting roles. The root:root 0755 versus root:docker 0750 policies and labctl traversal remain a separate unresolved issue. The legacy /etc/homelab:ro mount is preserved for labctl. No shared-parent chmod, Docker group membership, broader file access or credential rotation is part of the fix.

• New default/API file: /etc/komodo-deploy/komodo-deploy.env, 1001:1001 0600. Its dedicated parent is root:1001 0710, giving group search/traversal only, not listing/write. Renderer validation rejects incorrect owner/group/mode, extended/default ACLs and ancestor symlinks even for check/unchanged operations. Unsafe existing directories are not repaired.
• Enabled LAN preflight and actual runner use a same-path read-only directory --mount, which fails on missing source. Never bind the file inode; directory binding preserves visibility of atomic replacements.
• Explicit root-only --migrate-legacy verifies the existing old API pair equals reviewed SOPS before creating the new directory/rendering. It refuses a different credential at the new path or an unsafe existing directory. Old file, ciphertext and age key stay untouched; no rotation or recapture. CLI status: 0 unchanged/valid, 3 changed/would change, 1 failure. --check has no filesystem mutations and is not a migration dry-run.
• All three flags remained false in PR #508: common_komodo_deploy_api_reviewed, common_render_komodo_deploy_env, github_runner_komodo_deploy_api_enabled. A future cutover must enable them in reviewed persistent LAN-host inventory, not one-off extra vars alone, applying common provisioning before runner changes.

At that source-only checkpoint, the future controlled cutover requires separate approval. Pause new deployment dispatch and drain the LAN runner and independent writers first; verify fresh main and stage trusted merged code/ciphertext via the authorized host checkout process, preserving unrelated dirty and host-local files. Do not bootstrap through the broken relay. Run migration, then read-only validation, then the reviewed persistent opt-ins. Recreate only the drained LAN runner when approved, not Core/Periphery/Mongo or workloads. Preserve /opt/homelab/.git/komodo-deploy/, including lock and pending state.

Before bootstrap/workflow dispatch, verify the actual default UID/GID and supplementary groups (not a root or --user override), read-only same-path directory mount and current SOPS-matching credential with private read-only authentication. Repeat after both historical shared-parent modes and authorized convergence; deliberate alternation and atomic-replacement tests belong in isolated fixtures. A point-in-time helper read or successful migration is not convergence or application acceptance.

Rollback of a consumer retains the new mount and explicitly sets the existing KOMODO_DEPLOY_ENV_FILE=/etc/komodo-deploy/komodo-deploy.env. Reverting to old mounts restores the traversal bug, so dispatch stays paused. Legacy credential/bootstrap cleanup needs separate explicit approval and is never automatic migration cleanup. That #508 source-only pass performed no migration, inventory enablement, runner recreation, credential rotation, bootstrap or deployment. The later approved cutover checkpoint above supersedes its migration status, not its remaining acceptance gates.

Guided preparation — 2026-10-26 UTC

• Cached ghcr.io/moghtech/komodo-periphery:2.3.4 on infra-services, matching running digest fb4b52961fcce7a31e3776930e422967b6640f653de876f59f26dea243e29beb. Core/Periphery start times and running image IDs remained unchanged.
• Remote GitHub main is 7e7f79b6; the host checkout remains fec3dbc. Remote main is already explicit main dispatch only. The API workflow must preserve this policy, including case-sensitive Bash checks before checkout; no push trigger is restored.
• Live GitHub homelab-production has empty protection rules and no deployment branch policy. main reports unprotected; ruleset APIs returned HTTP 403 with a plan-upgrade requirement. Native reviewer approval was an unverified assumption, now corrected. No protection, billing or visibility settings changed.
• Created non-admin Service User homelab-deploy (6adf0356a10ff54a58019c59) through supported Komodo UI. Saved grants were re-read: 38 total — 36 Read, 2 Execute, only Sync infra-services-01 and Procedure deploy-infra. There are no specific permissions; base permissions are empty and server/build creation is off. New Procedures need Read grants after bootstrap.
• The owner created the Service User's API pair and captured it with the hidden-input helper. Local services/komodo/deploy-api.sops.yaml passed ciphertext-only validation; no plaintext credential was displayed or stored in the checkout. Preserve that existing encrypted source rather than minting another pair for rollout.
• Staged only renderer/ciphertext under root-private /etc/homelab/komodo-deploy-bootstrap-20261026/, outside the live checkout. The host age identity decrypted it. A root-only temporary candidate was checked with read-only /user, ListPermissions and GetPermission calls: enabled homelab-deploy, non-admin/non-super-admin, empty base permissions, 36 effective Read levels and exactly the two approved Execute entry points.
• Atomically rendered /etc/homelab/komodo-deploy.env as UID:GID 1001:1001, mode 0600. Ciphertext/runtime consistency passed. The running gha-homelab-lan-01 container, explicitly as UID:GID 1001:1001, could read it and authenticate through read-only /user. The temporary plaintext candidate was removed; root-private staging retains only renderer/ciphertext material.
• The host checkout remains fec3dbc; Core/Periphery start times and images are unchanged. No Sync, Procedure, deploy or restart was needed for credential verification. At that checkpoint, declarative Ansible enablement, code merge, ten new Procedure Read grants, reviewed bootstrap and application acceptance remained pending. The later extension increases the desired inventory/grants as recorded below; no additional live grants were applied.
• v2.3.4 permission review found that Sync and parent Procedure use internal administrative identities for their work. The caller needs Execute on the two entry points, not blanket downstream Write/Execute. Reviewed definitions remain a trust boundary; direct denial on a child cannot sandbox a changed parent.

What changed

• At the initial uplift stage, resources.toml defined 20 Procedures (now 26 plus 2 Actions in the source extension below). The recursive general graph selects each of the 19 automatic stacks once, conditionally. Komodo/OneUptime exclusions and the exact private canary remain intact.
• LiteLLM, the incident adapter, Paperless–Stirling bridge, Firefly, Orders and PR Dashboard reuse the private keyed fingerprint engine. No secret environment bytes are exposed as native tracked files. Dedicated force paths remain; new local-stack retry paths are explicit and unscheduled.
• LiteLLM copies non-secret config.yaml into its built image. A config-only change changes the proxy image rather than retaining a stale single-file mount or forcing database recreation. Its shared validated secret-rendering and PostgreSQL backup/recovery contracts must survive runtime integration.
• General deployment uses the native API and waits on exact Sync/Procedure Update IDs. A fixed delay, HTTP acknowledgement, missing resource or unknown outcome cannot count as success. New resources require reviewed external bootstrap and Read grants before the relay can see the complete declared inventory.
• Isolated SOPS API-credential rendering and runner validation are opt-in, preserve existing credentials on pre-replace failure, and add no runner Docker socket.
• Validation traverses nested Procedures, checks same-host pull ordering, rejects disabled/unreviewed execution shapes, and covers the corrected shell guard.
• Docs preserve newer-main owner-gated bootstrap, fail-closed secret rendering, coordinated database/shared-secret rotation, Periphery-only --no-deps recovery, LiteLLM logical recovery and Paperless uncertain-outcome recovery. The socket checklist now points to canonical maintenance instead of a generic Compose retry.
• Initial uplift navigation retained existing newer-main entries and added only the capability audit and two Komodo journals; the extension also links its image-update runbook. Manual-only dispatch is retained throughout; the GitHub environment name is not presented as a native approval gate.

Image-update extension — 2026-10-26, not live

A later source-only addition in the isolated PR #506 clone brings resources to 26 Procedures + 2 Actions, with 6 Repos and 21 Stacks unchanged. It adds six Procedures to the earlier 20; against the original ten live Procedures this is now sixteen new Procedures plus two Actions. No bootstrap, new grants, image-update token creation, Core recreation, pin write, PR publication, merge or deployment occurred in this documentation pass. The existing deployment API preparation above remains real and unchanged; it is not evidence that image-update credentials are provisioned.

flowchart TD
    N[Daily native same-tag check] --> R[Safe results or incomplete failure]
    O[Manual registry check or confirmed proposal] --> A[Inline Komodo Action or manual GitHub dispatch]
    A --> C[Bounded registry metadata checks]
    C --> P[Dedicated App creates draft PR when requested]
    P --> H[Human diff review and ordinary PR CI]
    H --> M[Human merge]
    M --> W[Separate explicit deployment workflow]
    W --> S[Exact successful external Sync]
    S --> D[Chosen cohort pull then conditional whole-stack reconciliation]

• check-image-updates owns daily 03:00 America/Los_Angeles, invoking its inline Action for 14 fixed native checks. Only same-tag digest changes are checked; digest-pinned/local-build images are unsupported and incomplete accounting fails. Native false means not detected or unknown, never current. Legacy Global Auto Update remains manual with schedule disabled and skip_auto_update = true. No Procedure enables auto_updates = true.
• Manual check-image-versions → request-image-pin-updates → GitHub check is report-only. Manual propose-image-pin-updates carries the exact PROPOSE_IMAGE_PINS confirmation in its definition and requests GitHub propose, never auto-merge/deploy/force. The Procedure is not an additional interactive prompt. Weekly Dependabot is unchanged; this complements it on demand, not a major resolver.
• Registry policy covers 13 image fields in six stacks across edge (Traefik, Homepage, AdGuard), documents (Paperless, Stirling) and finance (Firefly). It selects same-minor patches with prefix/suffix preserved. Major aliases use same-tag digest-only refresh, with no semantic patch guarantee. Immutable OCI index/manifest digests are verified; private registries, local builds, digest-only/unsupported refs, errors and truncation fail closed rather than producing false up-to-date claims or partial pin edits.
• Only allowlisted Compose image fields, literal Quick Reference Image table refs and future services/komodo/image-update-history.md entries may change. Unmatched docs warnings need humans; no arbitrary docs rewrites. History is header-only now, with no fabricated proposal/deployment entries. Rexit, OneUptime, Komodo and local source updates are excluded from the writer; private Rexit remains native-check-only with safe unknown metadata permitted.
• Both .ts Action sources are generated into inline resources.toml by scripts/render-komodo-actions.py; CI checks --check. Core has no code-checkout mount and must never execute runner-writable paths. The optional Core env file is absolute /etc/homelab/komodo-core/image-updates.env, root-owned parent 0700/file 0600, rendered by scripts/render-komodo-image-update-env.py from future SOPS ciphertext; only the example exists. The Core PAT needs repo Actions:write, not Contents:write. Its authority is broader than one workflow; fixed reviewed code enforces the whitelist. Provisioning and controlled Core recreation remain pending. Manual GitHub check needs no Core token or publisher credentials.
• The publisher requires a dedicated GitHub App installed only on this repo with Contents:write and Pull requests:write, vars.IMAGE_UPDATES_APP_CLIENT_ID and secrets.IMAGE_UPDATES_APP_PRIVATE_KEY, all unprovisioned. There is no GITHUB_TOKEN fallback so App-created draft PRs trigger ordinary CI. The Action supplies a UUID request_id, verifies exact Komodo image updates / op / cohort / uuid correlation within 35 minutes and never blindly retries an unknown outcome.
• The existing result-aware deployment workflow offers default deploy-infra plus deploy-reviewed-images-{edge,documents,finance}. Each new root pulls homelab before an exact conditional batch, never direct UI deployment. Cohorts reconcile whole changed stacks on current main, including non-image changes: review all relevant diffs and drain independent writers, not just review the image PR.

The live relay identity still has 38 effective grants: 36 Read + 2 Execute. Desired post-bootstrap access is 56 = 51 Read + 5 Execute over 26 Procedures, 6 Repos, 21 Stacks, 2 Actions and 1 Sync. The three new Execute grants are only the cohort roots; both Actions and all other new resources need Read. Check/propose Execute belongs to separately authorized operators, not the deployment bot. External reviewed bootstrap is required before the fail-closed relay can see the new declarations.

Metadata-only public-registry smoke checks passed for Docker Hub/Traefik, GHCR/Homepage and the custom Stirling registry, including manifest digest verification and bounded tag selection. No image layers, pin files, draft PRs or live deployments were created by those checks.

Integrated extension validation:

• Full Python CI-equivalent suite on Linux Python 3.13.17: 2,026 tests, 0 failures, 0 errors, 24 opt-in/tool-dependent skips. Node-backed Action tests and real Gitleaks boundary regressions ran. The disposable snapshot ran as UID 1001 with no network, host mounts, credentials or Docker socket during validation.
• Gitleaks 8.24.3 with full redaction: zero findings over the original PR history and the new staged snapshot, including new files. Four synthetic fixture findings are covered only by exact file-and-value AND exceptions for generic-api-key; regression tests preserve every other detector and reject broadened exceptions.
• Fixed the CI-specific subprocess loader failure by retaining LD_LIBRARY_PATH in the isolated Python fixture environment, not by exposing ambient credentials.
• The first extension CI run exposed missing node on the self-hosted runner PATH. Both Python/Komodo jobs now explicitly provision Node 22 with the pinned setup action; a regression test requires setup before Action tests rather than allowing silent skips. The corrected GitHub run must complete separately.
• Updated older flat-graph/webhook/static-hash assertions to preserve meaningful dispatch, exclusion, schedule, credential-isolation and source-integrity contracts.
• Inline Action drift and resource validation passed: 26 Procedures, 2 Actions, 6 Repos and 21 Stacks. Changed workflow/Compose YAML checks passed.
• Markdown passed for 220 pages; strict MkDocs and source/built documentation coverage checks passed, including the new workflow's project-map entry.
• Independent security review's two findings were corrected: the Core dispatch token is outside the recursively chowned checkout, and Action bodies are imported inline rather than executed from runner-writable host files. Follow-up review identified no remaining high-confidence blocker.

These local results do not claim that the new PR head's GitHub checks or live acceptance have completed. Owner gates and safe credential handling are in the new runbook.

Execution graph

flowchart TD
    R[Operator-reviewed explicit main dispatch] --> S[External Sync - wait for exact success]
    S --> P[deploy-infra]
    P --> I[infra-services chain]
    P --> X[Phoenix pull then conditional deploy]
    P --> Q[SonarQube pull then conditional deploy]
    I --> H[Homelab pull - render and fingerprint]
    H --> B[Conditional batch including Firefly]
    B --> A[Independent application children]
    A --> D[Reddit pull and conditional deploy]
    A --> O[Orders pull and conditional deploy]
    A --> G[PR Dashboard pull and conditional deploy]

A failed child does not stop parallel siblings, but still fails the parent. Preparation failures stop that host rather than deploying stale input. The relay lock does not serialize independent UI, scoped webhook or Ansible writers; drain those callers during general deployment and bootstrap.

Validation and remaining gates

Historical initial uplift validation, before the image-update extension above. Integrated validation ran in the isolated feature/komodo-deployment-uplift worktree based on 7e7f79b6f7cae45d98baf4e68196bcac55b18bdb:

• Resource validation passed: 21 Stacks, 6 Repos, 20 Procedures.
• The combined Komodo, relay, application, WS5 operating-contract, documentation and WS2 secret-rendering/procedure suites ran 456 tests successfully, with 2 opt-in checks skipped. Newer-main Firefly's default renderer backend and the owner-gated outbound-agent recipe remain protected by updated tests.
• All 4 safe-pull tests passed on Linux/Bash 5 in a disposable Python 3.13 container with the isolated checkout mounted read-only. No live host was used.
• Markdown fix/check passed for 219 pages; mkdocs build --strict passed; scripts/check-docs.py --site-dir site passed source and built-site checks. The original checkout's duplicate journal artifact was not copied into the PR.
• Changed YAML and both Ansible task files passed lint; the existing runner-task line-length and lint-configuration notices remain warnings, not new defects.
• Ciphertext-only validation passed and the encrypted API source matches the owner's artifact byte-for-byte. No local decryption was performed.
• git diff --check passed. Source-hash verification confirmed that the original implementation files were unchanged by packaging. No inventory, backup/restore implementation, canonical secret renderer, safe-pull implementation or PLAN.md changes were introduced.

The current-main LiteLLM renderer is used with required-key and URL-password validation and is included in the fingerprint. Integration did not revert the newer secret-rendering safeguards. PR CI and live application acceptance remain separate gates; local tests do not authorize merge or deployment.

Focused validation should cover input/secret mutations and unchanged selection, private-state failures, validated LiteLLM rendering, ownership-hook failures, exact Update identity, timeout/ambiguous outcomes, permission-filtered list responses, graph cycles, unsafe/disabled targets and host/application failure isolation. Run the existing operational-document contracts too; preserving them is part of integration, not an invitation to weaken their safety assertions. Safe-pull tests need Bash 4+ and Git. Source tests cannot substitute for live API execution or application acceptance.

See the rollout runbook for the three opt-in variables, pinned helper cache, reviewed external Sync, new-resource grants and first-run acceptance. Credential capture, dedicated-path migration, temporary LAN-identity preflight and persistent activation review/merge and the controlled runner apply are complete. Actual-runner read-only checks have now passed in both naturally occurring shared-parent states and after a full subsequent apply. External bootstrap and grants are now verified; first deployment and application acceptance remain pending. Do not dispatch until those gates pass. Native environment reviewer approval is absent, and no deployment is inferred from offline tests or version inspection.