Komodo explicit deployment (result-aware LAN relay)

This page retains its original URL. General deployment now uses Komodo's native API, not a pair of fire-and-forget webhooks. PR #506 merged as 6d6ff2e; production bootstrap is complete; first deployment/application acceptance remains pending. The credential-path correction is merged as PR #508 at 6ab7660 (26 checks passed). Credential migration and approved LAN runner activation are complete. The owner merged PR #510 as 0416d8e with all 26 checks passed. The activation used /opt/homelab at that reviewed merge; the later bootstrap preflight found 8bf7aa5 (#511), with no checkout change by this preflight. Preserve newer reviewed work and unrelated host edits. The unchanged credential matches approved SOPS and the retained legacy file. Actual-runner identity, read-only mount, credential access and non-admin authentication passed in both naturally occurring shared-parent modes and after a full subsequent apply. Credential/convergence, input preparation, external bootstrap and grants are complete; first deployment/application acceptance is not.

Current quarantine (2026-10-27 10:49:47 UTC): komodo-deploy.yml remains disabled_manually. Exact-run job admission safely replaced the historical broad LAN drain for unresolved edge run 40469274207. GitHub still reports queued with zero jobs; earlier cancellation/force-cancellation returned 409 and rerun returned 403. No new cancel/rerun retry, redispatch, edge execution or Sync occurred. Only the owned drain marker was removed after guard tests, with its backup root-private. LAN service is active; actual container runs as 1001:1001 plus group 10002, correct job-start hook/read-only mount and no Docker socket. GitHub verifies homelab-lan-01 online; single non-deploying qualification run 40491873440 on main 54ca4da and exact LAN job 122148472245 both completed SUCCESS, verified via GitHub API, including Set up runner and Run Linux and macOS qualification checks. Normal LAN job execution is restored. ansible-pull-check.timer and ansible-pull-apply.timer are active; MAINTENANCE, relay pending.json and drain are absent. Retain the hook and fail-closed pre-start verifier after restoration; removal needs terminal/no-executable-pending-work proof plus deliberate reviewed follow-up. See the guard contract. Merged #510's persistent flags remain applied only on infra-services; role defaults stay false. Phase: edge_dispatch_blocked_github_queue. Reddit's fingerprint was reconciled with the existing key; all seven input checks passed. External Sync Update 6ae04f5aa10ff54a5801dc65 completed successfully at reviewed source 824c148, leaving no pending resource changes/deployments. Live inventory is 26 Procedures / 2 Actions, and effective grants are exactly 51 Read + 5 Execute. No workload deployment or Core/Periphery recreation was performed. See the completed bootstrap record.

Fleet operations recovered separately: the LAN hold had also blocked dispatcher validation. RUNNER_FLEET_API_RUNS_ON now selects exact general runner homelab-ci-02; read-only health run 40485555074 completed successfully on that runner. Deployment and host-maintenance selectors were not changed. This temporary routing restores fleet API access without releasing the edge orphan or claiming all fleet health findings are resolved. It remains unchanged after LAN restoration, is not always-on redundancy, and watchdog apply remains guarded mutation. See API routing recovery.

Health limits (10:53:25 UTC): 49 running including the restored runner, 38 healthy, 11 without healthchecks, zero unhealthy/restarting; earlier DNS/TLS smoke good, not authenticated data acceptance. OneUptime and adapter October 27 B2 gaps are closed by explicitly owner-approved exact-snapshot copies; see the completed catch-up receipt. PR #520's local-retention fix is merged, but affected live units still lacked --retry-lock=5m at the 11:55 preflight. Normal convergence and next scheduled-run validation remain pending; no forced apply, restore/retention acceptance or broader mutation permission. Image check safely failed closed at 13/14 stacks / 32 services (Rexit latest exporter and studio versus deployed studio; native prefers deployed_services, not an auto_pull issue). Gmail Docker provisioning and the historical aiProject August 19 ghost are separate issues, not caused by this recovery. Normal work may resume; no mass restarts or replay of unknown canceled work. See health limitations.

Input preparation completed (05:25 UTC): the five authorized profiles passed native render/check and private credential compatibility gates. All six rendered outputs, including Orders' OAuth JSON, remained byte-identical; new private HMAC state is valid. Application containers and protected LiteLLM/Reddit state were unchanged, with timers restored. The earlier #513 source/re-preview gate remains complete. The initially deferred Reddit baseline was subsequently reconciled under separate approval, followed by the 18-create/13-update Sync and narrow grants. The historical preparation record remains separate from first deployment/application acceptance.

Historical post-merge failure (2026-10-26 UTC, before migration): manual run 40299611768 failed at credential loading, before host fast-forward, Sync, Procedure execution or new pending intent. The LAN runner could not traverse /etc/homelab while Ansible set it to root:docker, 0750; during that active apply it changed to root:root, 0755, and access returned. The legacy API file remained correctly 1001:1001, 0600. The follow-up moves the Komodo dependency out of this shared parent; it does not resolve the cross-role permission conflict or labctl access. Do not regenerate the key, chmod the shared parent or add the runner to Docker. Keep deployment dispatch paused until the separately approved cutover is accepted. See the checkpoint. Follow the bootstrap gates before enabling the new workflow. The audit records the previous production state separately. At that historical capture, the image-update extension was source-only: 26 Procedures and 2 Actions, including three additional approved deployment targets. Its bootstrap has since completed; Core dispatch token and publishing App provisioning remain pending.

Guided preparation update (2026-10-26 UTC): GitHub main 7e7f79b6 already uses explicit manual dispatch; the local API workflow preserves that newer policy. At that earlier capture the host checkout was fec3dbc; post-merge inspection found 034fe1d, still before the merged uplift. Live homelab-production has no native reviewer protection rules or branch policy, and main reports unprotected; ruleset API access returns a plan-upgrade requirement. An environment name is not an enforced approval. No GitHub settings, billing, or repository visibility were changed. Keep deployment manual-only and review the commit/CI before dispatch.

Historical credential checkpoint (2026-10-26 UTC, before the path fix): owner capture is complete. Ciphertext was validated locally, then staged with the renderer under root-private /etc/homelab/komodo-deploy-bootstrap-20261026/ without modifying /opt/homelab. The host render at the then-current path /etc/homelab/komodo-deploy.env was 1001:1001, 0600; read-only checks confirmed the expected non-admin identity and all 38 current effective permission levels: 36 Read and Execute only on Sync infra-services-01 and Procedure deploy-infra. That earlier runner read/auth check passed only at that point in time; the post-merge directory conflict above prevents treating it as stable. No Sync/deployment was invoked; Ansible/runner opt-in settings and new-resource grants remain pending. The temporary plaintext candidate was removed; retain the bootstrap renderer/ciphertext until the matching merged source and declarative provisioning are verified.

Why not a GitHub repo webhook?

*.infra.realemail.app resolves on the LAN. GitHub cloud deliveries cannot reach Komodo directly. The managed LAN runner is still the bridge; no public ingress, runner Docker socket, or Core database access is added.

The five-minute resource poll refreshes state and pending changes. It does not execute Sync or deployment. Missed-trigger recovery is deliberately manual through the approved workflow; no recurring full-deployment schedule is introduced.

Architecture

sequenceDiagram
    participant GH as Operator-reviewed explicit main dispatch
    participant R as LAN runner
    participant K as Komodo API
    GH->>R: komodo-deploy.yml
    R->>R: Credentials, main ref, shared lock and pending-state recovery
    R->>K: Verify managed-resource visibility and idle state
    R->>R: Safe host fast-forward under lock
    R->>K: Recheck idle state, execute RunSync
    K-->>R: Exact Update ID
    R->>K: GetUpdate until Complete and success
    R->>K: Rediscover resources and recheck idle state
    R->>K: Execute RunProcedure deploy-infra (default, or selected cohort)
    K-->>R: Exact Update ID
    R->>K: GetUpdate until Complete and success
    R-->>GH: Verified result, or fail with pending state preserved

Piece	Contract
Workflow	.github/workflows/komodo-deploy.yml; manual main dispatch only, case-sensitive confirmation before checkout, shared concurrency; environment label does not enforce review
Relay	scripts/trigger-komodo-deploy.py; HTTPS native /execute and /read, no credential-forwarding redirects
Credential (migrated; actual-runner/convergence acceptance complete)	/etc/komodo-deploy/komodo-deploy.env; UID:GID 1001:1001, mode 0600, two API values only; parent root:1001, 0710, group search-only
SOPS source	services/komodo/deploy-api.sops.yaml; operator-created encrypted file, separate from Core secrets
State	/opt/homelab/.git/komodo-deploy/; private persistent lock and pending.json
Deadline	KOMODO_DEPLOY_TIMEOUT_SECONDS, default 900 seconds, inside the 20-minute workflow bound
Sync	infra-services-01, linked to homelab, services/komodo/resources.toml; managed/write-back and delete remain off
General Procedure	deploy-infra; internal/general webhooks disabled; scoped release/retry paths remain separate
Approved target choice	Default deploy-infra, or deploy-reviewed-images-edge, deploy-reviewed-images-documents, deploy-reviewed-images-finance; same lock/Sync/exact-Update checks for every root

There is no push trigger. Merges, documentation changes and runtime changes do not automatically dispatch general deployment. Path filters are therefore unnecessary. The only event is workflow_dispatch with exact refs/heads/main and DEPLOY_HOMELAB, checked in Bash before checkout because GitHub expression equality is case-insensitive. This operator-intent gate is not equivalent to branch protection or an independent required reviewer. It also does not cancel already queued work. The target input is separately checked against the four exact approved roots before checkout and by the relay. No arbitrary Procedure name or direct UI deployment bypass is supported. The diagram shows the default graph; a cohort uses the same ordered external Sync and selected-Procedure result verification.

Managed LAN runner (infra-services)

DEPLOY_RUNS_ON selects the runner. It needs LAN DNS/reachability and the existing /opt/homelab:rw checkout. With github_runner_komodo_deploy_api_enabled: true, both the preflight container and the actual LAN runner receive the same directory bind: --mount type=bind,src=/etc/komodo-deploy,dst=/etc/komodo-deploy,readonly. --mount fails if the source is missing; do not replace it with an auto-creating volume bind. Bind the directory, not the file; do not bind an individual credential inode. This keeps atomic credential replacement visible without pinning an old inode. The enabled launcher validates the API file before registration; validation in a helper alone does not prove the actual runner mount.

The parent is root:1001, 0710: group 1001 can search/traverse, but cannot list or write it. The API file remains 1001:1001, 0600. The renderer validates the parent's root owner, group, exact mode, absence of extended/default ACLs and absence of symlinks anywhere in its directory path, even for --check, --validate-existing and an unchanged render. Unsafe existing directories are rejected, not repaired.

The legacy /etc/homelab:ro mount remains for labctl. Its conflicting root:root 0755 / root:docker 0750 role policies and labctl traversal remain a separate, unresolved issue. This fix neither changes those roles nor grants Docker group membership or a Docker socket to the LAN runner.

The relay's flock spans recovery, host fast-forward, Sync and deployment. All relay callers must use the same host state directory. This lock does not cover independent UI, release-webhook, direct Compose or Ansible checkout writers. Drain those callers before the run and keep them drained until it finishes. Native busy-state checks are fail-closed preflight checks, not an atomic distributed lock.

Rollout and bootstrap

Do not dispatch the general workflow until these prerequisites are complete. The explicit webhook-disable guard is merged in #513 and its pending source was revalidated. Preserve explicit false on the four new operator-only retries: deploy-litellm, deploy-oneuptime-incident-adapter, deploy-paperless-stirling-bridge, and deploy-firefly. Omission is enabled in v2.3.4; preserve the six legacy scoped webhook contracts rather than claiming all webhooks are disabled. Native environment approval is currently absent; do not wait for an approval screen that does not exist or describe the workflow as protected by one. An operator must coordinate the maintenance window; no credentials are minted by editing Mongo, and no password or API secret belongs in a command line or job log.

1. Drain first, then stage trusted merged code. In a separately approved maintenance window, pause new deployment dispatch and drain the LAN runner, queued workflows and active Sync/Procedure/release work. Coordinate independent UI, Compose and Ansible writers. Review fresh main and CI, including the merged credential fix, then use the authorized host checkout process to stage that code and reviewed ciphertext on /opt/homelab, preserving unrelated dirty and host-local files. Do not bootstrap through the broken relay, use an unmerged checkout or broadly reset/stash/clean host work. Preserve /opt/homelab/.git/komodo-deploy/. Follow the credential-path cutover before bootstrap. Native GitHub protection is absent; preserve manual-only dispatch. Do not resume paused Reddit placement work as a validation shortcut.
2. Versions and helper images. All four inspected components (Core, local agent, Phoenix and SonarQube agents) were v2.3.4. Source now pins matching :2.3.4 images. Pre-cache that exact Periphery image before using --pull never helpers; the initial audit found only floating-tag images cached. Guided preparation has now cached :2.3.4 at digest fb4b52961fcce7a31e3776930e422967b6640f653de876f59f26dea243e29beb, matching the running agent. No containers were recreated. Apply Compose pins outside Komodo, using self-managed maintenance. See LXC version evidence and maintenance.
3. Use the dedicated non-admin Service User. homelab-deploy was created via Settings → Users → New Service User, leaving UI-write disabling and OIDC settings unchanged. Its historical 38 grants were Read on 36 resources and Execute only on deploy-infra and infra-services-01; base permissions are empty, server/build creation are off, and no specific terminal/log/inspect grants were added. The earlier uplift added ten Procedures; the image extension adds six more and two Actions. Approved bootstrap now verified 56 effective grants = 51 Read + 5 Execute across 26 Procedures, 6 Repos, 21 Stacks, 2 Actions and the existing Sync. Execute remains on the original two entry points and adds only the three deploy-reviewed-images-{edge,documents,finance} roots. Every other resource needs Read, including the check/propose Procedures and both Actions; operator check/propose Execute is not required by this bot. Execute is the effective level in place of Read, not an extra grant on the same resource. The old 38-grant checkpoint is historical; only 15 Read and 3 Execute additions were made. Keys inherit this user's permissions; they have no independent per-key scopes.

In v2.3.4, native Sync and Procedure execution check the entry-point Execute grant, then use internal administrative identities for resource writes/nested executions. The relay user does not need downstream Write/Execute grants. This makes reviewed Git/Sync/Procedure configuration a critical trust boundary; denying direct Execute on OneUptime/Komodo alone cannot sandbox a changed parent. Review inherited/everyone grants and effective access after credential capture. 4. Credential capture is complete; do not recreate it for rollout. Preserve the existing encrypted source and the owner's expiry/rotation record. The following is the fresh-identity capture procedure, not an instruction to overwrite the captured file. For a future owner-authorized new credential, open the prepared Service User, then API Keys → New API Key (not your personal Profile). Name it homelab-deploy-relay, choose a finite expiry such as 90 days, and create it. The Secret is shown once. In a terminal at this repository root run:

uv run python scripts/bootstrap-komodo-deploy-api.py

Paste Key and Secret into the hidden prompts. The helper sends plaintext only through SOPS stdin and publishes only encrypted services/komodo/deploy-api.sops.yaml, refusing existing files/symlinks. It never deploys, renders runtime credentials, or overwrites Core secrets. Do not paste either value into chat, shell arguments, screenshots or logs. Store the credential and expiry in the owner's approved password manager, close the one-time-secret dialog, and report only that encryption completed. Expiry/rotation remains owned by the operator. 5. Render before runner enablement: migration is complete on infra-services. The approved root-only --migrate-legacy operation and both read-only checks passed. Recheck the new parent/file and SOPS consistency in the next approved window; do not repeat capture or rotate the credential. The explicit migration procedure below remains the contract for an unmigrated host. The renderer preserves the existing file on pre-replace failure. Rendering requires SOPS, Python/PyYAML and the approved host age identity. The existing trusted group-readable age-key policy is supported; do not change the age key's permissions for this feature. 6. Enable runner validation: persist opt-in and validate the actual runner. Only after successful migration and validation in step 5, and separate cutover approval, record this future enablement in reviewed persistent LAN-host inventory, not only one-off extra vars:

common_komodo_deploy_api_reviewed: true
common_render_komodo_deploy_env: true
github_runner_komodo_deploy_api_enabled: true

All three enable/review variables default false in their roles. Merged PR #510 sets these exact overrides only in infra/ansible/inventory/host_vars/infra-services.yml. Owner review/merge and the narrow credential/runner applies are complete. Actual-runner checks passed in both parent states during natural convergence and after a full apply. This gate is satisfied; bootstrap/grants and deployment approval remain separate. Do not enable unrelated hosts. Common provisioning precedes the runner gate, which checks encrypted-source/output consistency before changing runner configuration. Recreate only the drained LAN runner when separately approved; verify its default identity/groups, read-only directory mount and current API authentication across parent-mode changes and convergence as specified below. 7. Prepare host inputs within the approved scope. From the reviewed host checkout, recheck encrypted inputs, clean external sources, native Python >=3.11, Git, SOPS, destination metadata and private runtime-credential prerequisites. The completed input-only window prepared only five missing profiles: oneuptime-incident-adapter, paperless-stirling-bridge, firefly, orders, and gh-pr-dashboard. As the authorized root host operator, use native /usr/bin/python3 -B /opt/homelab/scripts/komodo-deploy-inputs.py <stack>, then the same command with --check, serially and stopping on failure. Do not use the shell wrapper's Docker/nsenter fallback for this native-only window. LiteLLM and Reddit were read-only rechecks only. LiteLLM passed; Reddit was stale against #514's Compose change and was initially preserved. A later approved same-key fingerprint refresh reconciled it without secret rendering; all seven input checks then passed before external Sync. Do not repeat completed preparation. Reddit SOPS/rendered-env consistency remains distinct from fingerprint success; render Reddit secrets or refresh its fingerprint only under separate approval. Preserve private state; never delete dirty sources or secret files to pass a gate. A first fingerprint can select a later deployment, not authorize one now. 8. Complete a reviewed external bootstrap Sync while idle. Use the existing authorized administrator path to apply the new declarations and inspect terminal success. The relay requires every declared managed resource to exist and be readable; it will not infer whether a missing list result is absent or hidden by permissions. New resource additions need the same reviewed bootstrap step before the general relay can reconcile them. Confirm grants on the new child/retry Procedures and Actions, plus Execute on the three approved cohort roots. Check exact bootstrap schedules: the new check-image-updates Procedure owns daily 03:00 America/Los_Angeles; legacy Global Auto Update has its schedule disabled and retains manual skip_auto_update = true. Check disabled internal webhooks, inline Action source agreement and exclusions. No auto_updates = true is enabled. The image bootstrap separately covers the optional Core env and publisher credentials; neither is a prerequisite for a manual GitHub registry check. 9. Dispatch one workflow and inspect acceptance. Start only the intended reviewed main run. Complete the checks below before restoring normal caller/convergence activity. Record implemented, deployed and application-verified states separately.

Activation review and runner resume

Historical activation applied — 2026-10-26: after PR #510 review/merge and the normal apply's successful 12:01 UTC completion, the narrow credential and runner applies succeeded. Only the owned drain marker was released; no redundant restart was requested. At that capture the actual runner was online with its read-only API directory and default identity verified. The later 08:14 UTC October 27 broad hold was superseded by 10:49:47 UTC exact-run quarantine: service/container running, GitHub-online, qualification run 40491873440 successful; normal LAN jobs restored. Phase remains edge_dispatch_blocked_github_queue; no automatic redispatch.

The remaining observational gate was completed on 2026-10-26: the same actual runner passed at root:root 0755 at 18:26 UTC, during the naturally occurring root:docker 0750 state at 18:57 UTC, and after the full apply at 19:05 UTC. The apply exited 0 with a successful full recap; no forced apply, live chmod, credential rotation or runner restart was used to produce the evidence. The separate 17:50 UTC check-mode failure in operator-shell : Set operator shell home remains unresolved; credential acceptance is not an all-host-checks-passed claim.

Owner action next: reconcile run 40469274207 through inspection/GitHub support, not further cancel/rerun retries. Keep deployment disabled; no redispatch or direct Procedure execution. Do not remove the guard just because the runner is restored or qualifies: removal requires proof the orphan is terminal with no executable pending work plus deliberate reviewed follow-up. Bootstrap/grants need no repetition. See the guard contract.

The following is the controlled sequence used for activation, retained for recovery or a separately approved future rerun—not a request to repeat completed steps. Any future explicit runner-role apply may rewrite the instance environment: preserve and revalidate the job-start hook/pre-start verifier; never bypass them to start the service or treat this historical marker-release recipe as guard-removal approval.

1. Recheck fresh reviewed main/CI, workflow-disabled state, queued/running jobs, runner drain/offline status, relay pending state and independent writers. Record original control states; pause conflicting convergence/intake and let active backup/restore or Ansible work finish rather than interrupt it.
2. Advance /opt/homelab to the reviewed activation commit through the authorized checkout process under the appropriate locks. Preserve unrelated edits/local files, the private relay state and cutover archives. Do not reuse the earlier Rexit staging exception without fresh equality/conflict checks. If a bounded operation times out, inspect its actual outcome before retrying.
3. Verify effective persistent inventory from both generated.yml and github-runners.yml: all three flags true only on infra-services, with only homelab-lan-01 allocated there. Apply the narrowly scoped common credential task first and require --check / --validate-existing to return 0.
4. Apply only the github_runner role on infra-services with github_runner_instance_allowlist: [homelab-lan-01]. The stock runner playbook also imports github_runner_observability; use the reviewed private wrapper to exclude that unrelated role, not broad fleet maintenance or a full site apply. The ordinary site playbook does not apply the runner role. Keep the owned drain marker throughout configuration: systemd can start the new launcher, but it waits without creating/registering a container. After successful apply and fresh queued work/registration checks, remove only that marker. The waiting launcher continues without a redundant restart or adapter resume. Keep deployment disabled and do not recreate Core, Periphery, Mongo or workloads.
5. Verify the actual recreated container, using its default identity, not a helper or --user override: UID/groups, read-only directory mount, current SOPS-matching credential and private read-only API authentication. Repeat across authorized convergence and both naturally observed shared-parent states. Never chmod the live parent or rotate a key merely to test. If either state or other acceptance evidence is missing, record the gap and keep deployment blocked.
6. Restore only controls owned by this window to their recorded prior state when safe; do not strand routine convergence awaiting another review. Record exact acceptance evidence. Keep deployment disabled until the separately reviewed ResourceSync/new-resource grants and deployment gates are satisfied.

The native drain's 30-second stop timeout was followed by a confirmed stopped container (service exit 137); it did not produce a success replay receipt. The 90-second generic safe-pull timeout did not advance HEAD; the later targeted fast-forward succeeded. Neither timeout justifies blindly replaying an operation. The later 90-second registration wait also expired, but follow-up found the new runner online with zero service restarts; registration was not retried. The journal retains the evidence and preservation details. The old Ansible check exit 2 predates cutover; a later exit 0 can include a lock skip. Restored timers alone do not establish all-checks-passed convergence.

Credential-path cutover (separate approval required)

This is the controlled-window procedure, not authorization to bypass activation review. On infra-services, migration, post-migration checks and persistent activation review/merge and runner provisioning are complete. Actual-runner read-only checks passed in both naturally occurring parent states and after full convergence. Bootstrap/grants and deployment acceptance remain separately gated. Keep new deployment dispatch paused throughout; drain the LAN runner and independent writers before staging a fresh reviewed, merged main on the host. Use the existing authorized checkout process without discarding unrelated dirty files or host-local secrets. Check the exact renderer/ciphertext revision on the host; the older root-private bootstrap renderer is historical, not the new migration code. Do not ask the credential-blocked relay to update itself. Preserve the entire /opt/homelab/.git/komodo-deploy/ lock/pending-state directory, even though the 2026-10-26 failed run created no new intent.

1. Verify prerequisites without recapture. Require Linux, Python >=3.11/PyYAML, SOPS, the approved age identity and the existing legacy API file. Review services/komodo/deploy-api.sops.yaml; never print plaintext or substitute the example. Do not pre-create/chmod the destination or use an ordinary render to bypass the legacy-pair comparison.
2. Run the explicit migration as root from the trusted merged host checkout:

sudo python3 /opt/homelab/scripts/render-komodo-deploy-env.py --migrate-legacy

This fixed-path mode first verifies the existing /etc/homelab/komodo-deploy.env pair and its 1001:1001, 0600 metadata against the reviewed SOPS pair. Only after equality is established may it create /etc/komodo-deploy as root:1001, 0710 and atomically render the same pair as /etc/komodo-deploy/komodo-deploy.env, 1001:1001, 0600. A missing/mismatched legacy pair, different credential at the new path, unsafe existing directory, extended/default ACL or ancestor symlink fails closed. A valid identical new credential is unchanged. The old file, ciphertext and age key are left untouched; this is migration, not rotation or recapture. 3. Validate after migration, before runner changes. Run individually and inspect each exit status:

sudo python3 /opt/homelab/scripts/render-komodo-deploy-env.py --check
sudo python3 /opt/homelab/scripts/render-komodo-deploy-env.py --validate-existing

The renderer contract is 0 unchanged/valid, 3 changed (or would change with --check), 1 failure. Success is silent. Do not chain commands with && and mistake migration's expected 3 for failure; post-migration checks must return 0. --check performs no filesystem mutations, including directory creation, permission repair or temporary-file writes. It is not a migration dry-run and cannot be combined with --migrate-legacy; ordinary check fails on an absent destination directory. Common-role Ansible check mode can report an absent directory as planned creation but leaves it absent and skips output validation. --validate-existing checks the parent and private file without SOPS or an age key; it does not establish equality with the encrypted source or API authentication. 4. Persist the reviewed LAN-host settings for continuing convergence. In the separately reviewed cutover inventory, enable common_komodo_deploy_api_reviewed and common_render_komodo_deploy_env together with github_runner_komodo_deploy_api_enabled. Apply the narrowly scoped common credential task before runner maintenance. All three must remain enabled in the effective persistent inventory; one-off -e flags alone are insufficient. Merged PR #510 supplies infra-services-only overrides; role defaults remain false. Common creates only an absent directory; it refuses unsafe existing access. The runner gate requires a source/output check returning 0 before runner configuration changes. 5. Recreate only the drained LAN runner when approved. Preserve the checkout and private relay state; do not recreate Core, Periphery, Mongo or workloads for this path migration. Confirm the enabled preflight and actual runner both use the same read-only directory --mount. Inspect only needed mount/identity metadata, not environment dumps. Verify the actual container's default UID:GID 1001:1001 and supplementary groups, without root/--user overrides or adding Docker membership. On 2026-10-26 the supplementary group was 10002, not host Docker GID 987. Validate file access and privately authenticate the current SOPS-matching pair through read-only identity/permission calls. 6. Prove convergence independence before bootstrap or workflow dispatch. Repeat actual-runner identity, read-only mount and current-credential checks after authorized convergence through both existing /etc/homelab states: root:root 0755 and root:docker 0750. Use isolated fixtures for deliberate mode alternation and atomic-replacement checks; do not chmod the live shared parent as a workaround or rotate a live key merely to test replacement. A one-time read or helper-only preflight is not acceptance. Record both-mode and convergence evidence; if incomplete, keep deployment blocked. Once accepted, continue the reviewed external bootstrap Sync/grants and workflow/application verification gates above; credential acceptance alone is not deployment success.

Rollback and legacy retention

Keep dispatch paused during rollback. If reverting the consumer, keep the new read-only directory mount and explicitly set the already-supported KOMODO_DEPLOY_ENV_FILE=/etc/komodo-deploy/komodo-deploy.env in its reviewed persistent configuration. Revalidate default-runner access and current credentials before any resume. Reverting to an old-mount-only launcher restores the original traversal bug; it is not a working rollback, so keep dispatch paused. Do not compensate with shared-parent chmod, extra Docker groups or broadly readable files. Preserve private pending state. Legacy credential/bootstrap-artifact cleanup needs separate explicit approval after cutover acceptance; it is never automatic migration cleanup.

One-time setup (done 2026-06-26)

That date records introduction of the LAN webhook relay, not completion of this API migration. The old KOMODO_WEBHOOK_SECRET remains relevant to separately scoped release/remediation listeners. It is not a fallback for general deployment now.

Reddit Explorer scoped rollout

The existing Reddit fingerprint and scoped deploy-reddit-explorer force path are preserved. Its conditional path now lives in reconcile-reddit-explorer, after the ordinary infra-services batch. No cache/placement operations are added. Follow the shared bootstrap gates above and the application README's private-state policy.

• Verify the companion application's release webhook and any override select the scoped Procedure, not disabled deploy-infra; review application PR #100's integration rather than assuming its rollout.
• Drain old queued work and competing checkout/render writers. Preserve the private key/fingerprint, missing-input and dirty-source failures, and paused placement work. First adoption may deploy once. Use isolated fixtures, not production secret/cache mutations, to test change/failure detection.
• A successful Compose up can advance the comparison baseline before a smoke failure. Fix the cause, then deliberately use deploy-reddit-explorer for an unchanged-input force retry; inspect pull/deploy Updates and application health. The fingerprint is desired input state, not a success receipt or atomic build snapshot.

Manual trigger

The current edge run is unresolved; complete its queue-recovery gate and obtain a separately reviewed deployment decision before using this recipe. Guarded runner restoration or qualification does not authorize dispatch or guard removal. For a paused workflow, keep the approved run's workflow enabled through job creation/completion rather than immediately toggling it off after HTTP acceptance. A queued request is not a completed or cancelled run; never redispatch solely because no job is yet visible.

As an authorized repository operator:

1. Open Actions → Komodo deploy (explicit main dispatch) → Run workflow.
2. Independently review the intended main commit, CI result and rollout checklist.
3. Select main and enter exactly DEPLOY_HOMELAB. No native environment reviewer approval is currently configured; this explicit owner dispatch is the intent gate.
4. Choose the approved target: deploy-infra (default), deploy-reviewed-images-edge, deploy-reviewed-images-documents, or deploy-reviewed-images-finance. For cohorts, complete the image review and merge gates. Each pulls homelab, then conditionally deploys its exact stack batch. A cohort reconciles whole changed stacks on current main, including non-image changes, not just images or the proposal commit. Review all relevant diffs and drain independent writers; never invoke the cohort directly in the UI to bypass this path.
5. Require a successful, non-skipped job and complete Verify.

Feature branches, missing confirmation and missing credentials do not deploy. Do not run a host-side relay command to bypass operator review. Direct helper callers must supply exactly KOMODO_DEPLOY_REF=refs/heads/main; a missing, short, tag or feature ref fails before credential resolution or execution. This is not permission to bypass the workflow gates. The credential file is primary; only an absent file permits an explicit complete KOMODO_API_KEY / KOMODO_API_SECRET environment pair. Invalid or partial files fail closed. No signed-webhook downgrade exists.

Verify

• The relay reports successful exact RunSync and RunProcedure Updates. IDs originate from each execute response (_id.$oid), not a "latest Update" query. Complete without success = true is a failure.
• For a cohort target, confirm its successful PullRepo homelab precedes the exact conditional batch: edge traefik,homepage,adguard; documents paperless,stirling-pdf; finance firefly. Check all affected application/data behavior, including any non-image changes; a successful image proposal is not deployment evidence.
• For default deploy-infra, the top Procedure waits for its nested host chains. A failed child makes the parent fail, while other independent host chains finish. Inspect nested outcomes and application smoke/health checks; Compose success alone is not acceptance.
• No source/config/environment change should rebuild Orders/PR Dashboard or the six input-aware stacks on an unchanged second run. Source-only/config-only/secret-only changes must select the correct stack. LiteLLM config-only changes must recreate its proxy from a changed image without forcing its database lifecycle.
• Failed LXC pulls must not deploy stale LXC inputs or prevent infra-services work. Failed application source pulls must not suppress Firefly, which is in the earlier batch.
• Komodo stays self-managed; OneUptime stays manual-upgrade; canary scope is unchanged.
• The success summary is emitted only after the relay verifies success. A failed, cancelled or skipped step must not claim deployment completion.

Focused offline tests and graph simulation are part of the integration validation plan; their results must be recorded against the isolated PR revision. The earlier 2026-10-26 credential check was point-in-time evidence at the legacy path. Migration and temporary preflight alone did not establish actual-runner acceptance. The subsequent direct probes during both natural parent states and after a full apply now satisfy that gate. The first successful deploy-capable API run and application acceptance remain pending; reviewed bootstrap and grants are now complete.

Troubleshooting

Symptom	Action
Missing/invalid API pair	Complete the reviewed credential-path cutover and actual-runner gates; do not recapture a valid pair or fall back to the webhook secret
Parent directory or mount rejected	Require /etc/komodo-deploy as root:1001 0710, no extended/default ACL or ancestor symlink, and the same read-only directory mount in preflight and actual runner; do not repair /etc/homelab as a workaround
Resource missing or unreadable	Review API grants and bootstrap Sync; successful filtered list responses are not proof of absence
Resource busy / nonterminal Update	Inspect exact active operations and drain independent callers; do not repeatedly dispatch
Host lock held	Another relay owns the shared lock; wait for its verified result
Deadline / transport failure during execute	Outcome is unknown, not cancelled; follow pending-state recovery below
Source/secret input check fails	Preserve the failing inputs, fix render/source prerequisites, refresh and recheck; never fake a fingerprint
Pinned helper image missing	Pre-cache the reviewed :2.3.4 image in maintenance; helpers intentionally never pull
Stale Docker socket	Use socket recovery, then an approved workflow retry
Partial deployment	Inspect each child and data/migration gates; there is no automatic rollback

Pending execution recovery

The private journal persists intent before an execute request. A timeout or lost response is never blindly retried. Known Update IDs are re-read to terminal state on next invocation before host fast-forward or new work. A failed terminal Update stops that invocation; fix the cause before a deliberate subsequent retry.

If pending.json has no Update ID, is corrupt, or cannot be verified, stop. An operator must hold the same host lock, inspect Komodo Updates and correlate the exact operation, target and dispatch time before repairing the journal or clearing proven completed intent. Do not invent an ID, substitute the most recent Update, delete state because it is old, or treat idle action flags as proof an ambiguous operation never ran. If correlation cannot be established, leave the pipeline blocked.

The lock file is never unlinked: deleting it can create independent lock inodes and allow concurrent relays. Preserve the state directory across runner replacement.

Host-checkout conflicts (untracked or dirty tracked files)

git merge --ff-only refuses an incoming change that would overwrite an untracked path (including gitignored files) or dirty tracked file. Either can block subsequent deployment; see the July outage. The August 13 failure after PR #290 was the tracked-file case, even though host copies already matched the incoming revision.

The relay runs scripts/komodo-safe-pull.sh under its host lock, fetching from the Actions workspace rather than unavailable host SSH aliases. Before fast-forwarding, it copies incoming-path blockers under /opt/homelab/.git/komodo-untracked-backups/, records a diff when needed, and removes only the archived untracked blocker or restores the blocking tracked path to HEAD. Local edits not touched by the incoming commit remain. Inspect the archive before resolving conflicts; never use a broad reset/cleanup as a shortcut.

Manual checkout recovery must also coordinate the shared lock, pending-execution recovery and independent writers. Compare the exact incoming revision before touching files. Never recursively remove service directories: gitignored runtime secrets can live beside tracked files; deletion of LiteLLM's .env during cleanup caused the historical outage. Identify recurring writers—live tracked edits, manual tests or an interrupted checkout—rather than treating automatic archival as prevention.

After recovery, use the complete Manual trigger recipe and Verify; a skipped dispatch is not recovery.

Empty repo secret

Historical webhook fallback failure: see the July incident audit. General deployment no longer consumes the GitHub webhook-secret fallback. The rotation script still emits renderer status lines alongside the secret. Do not capture or pipe whole rotation stdout into a secret value. Its output contract and verified single-value transfer remain a separate source dependency; see coordinated database/secret changes. Never print secret values to diagnose authentication.

Webhook signature mismatch

This now concerns scoped release/remediation callers only. Compare credentials privately against the listener's running configuration without printing prefixes or full values. A webhook's HTTP 200 still does not establish completion; inspect the scoped Procedure's terminal Update.

Mongo password drift

MONGO_INITDB_* only applies on first empty-volume initialization. Rendering a new Mongo password or recreating Mongo does not change an existing database user's password. Do not recreate Core with mismatched credentials or dump container environments to recover them; initialization metadata is not proof of live credentials. Stop and follow the coordinated database/secret procedure, retaining recovery evidence and approved credential access. Never reset/delete Mongo volumes or undo only SOPS after a partial rotation. The isolated API file does not rotate Mongo, JWT or OIDC secrets.

After Komodo SOPS rotation or render

For Core secrets, follow render verification and consumer-specific maintenance before general deployment. Rendering alone does not rotate Mongo or reload running containers, and Core-only recreation is not a shared-secret rollout. Never run a full rotation just to retry authentication. Keep rotation stdout out of logs and update remaining scoped webhook consumers only through approved secret handling. For a separately approved rotation of the isolated API pair after cutover, provision an approved replacement through SOPS and atomically render it at /etc/komodo-deploy/komodo-deploy.env, validate runner access from the actual LAN container's default identity, then retire the old key only after a verified run. The read-only directory mount sees the replacement; no Core or runner recreation is required merely to expose that inode. Initial adoption of the new directory mount does require the approved drained-runner recreation above. Do not rotate or recapture credentials for this path migration.

After host-side acceptance and any pending-state recovery, use Manual trigger and inspect exact Updates and application health in Verify. Rendering or read-only authentication alone is not deployment acceptance.

Related

• Komodo service manual
• Capability audit and implementation status
• GitHub runner fleet
• Komodo upstream API