Komodo capability and efficiency audit — 2026-10-25¶

Verdict¶

Good foundation; not yet making full use of Komodo's useful deployment features. The highest-value work is dependency-aware deployment, verified execution ordering, and failure isolation—not additional alerts, more dashboards, or enabling every feature. Keep Komodo; improve how the existing resource model is used.

The original assessment below records the captured production baseline, not a completed rollout. The owner subsequently authorized all high/medium findings; the implementation status below supersedes their original recommendation status. No workload deployment has been performed for this uplift. Historical guided preparation cached the pinned helper image, created/scoped non-admin homelab-deploy and verified the SOPS-encrypted API pair through read-only calls and the then-running LAN runner at the legacy path. The first deploy on 2026-10-26 UTC subsequently failed at credential loading before mutation during shared-parent permission changes. PR #506 (6d6ff2e) and the credential correction #508 (6ab7660) are now merged; the approved migration retained the existing pair unchanged. #510 review, controlled activation and scoped credential/both-mode/natural-convergence acceptance completed at 2026-10-26 19:05 UTC. Current phase is edge_dispatch_blocked_github_queue: the owner-approved edge attempt remains queued before job creation (see current checkpoint below). The authorized Reddit same-key fingerprint refresh, all seven input checks, external Sync and least-privilege grants completed on 2026-10-27 UTC and remain complete. Configuration bootstrap is complete, not workload rollout or application acceptance; the separate operator-shell check-mode failure remains unresolved. See the bootstrap completion record and activation procedure. The earlier remote-main baseline 7e7f79b6 introduced manual-only dispatch; this implementation preserves it. Live GitHub environment review/branch protection was found absent, not assumed configured.

Implementation status after owner approval¶

The table describes the source uplift integrated onto 7e7f79b6 and merged as 6d6ff2e, with configuration bootstrap completed from reviewed main 824c148 on 2026-10-27; workload rollout is not complete. The journal records integration/CI, migration/activation and bootstrap evidence. The manual GitHub registry-check path succeeded independently of Komodo bootstrap.

Finding	Implemented in source	Deployment / acceptance
1: Actual deployment inputs	Six scoped private HMAC profiles; render-before-fingerprint; native tracked file; predeploy verification; local builds explicit; LiteLLM config baked into image	Five-profile preparation passed with six byte-identical outputs; subsequent same-key Reddit refresh and all seven input checks passed. Input declarations are live; real changed/unchanged application runs pending
2: Result-aware relay	Exact Sync/Procedure Update completion, private credentials, persistent intent and host lock, full managed-resource visibility, busy checks, main-only truthful workflow	#508 credential migration unchanged; #510 activation and scoped credential/both-mode/natural-convergence acceptance complete. External Sync/grants complete; successful deployment remains pending. Not whole-host check-mode acceptance
3: Selective apps	Orders/PR Dashboard conditional; Firefly conditional after rendering; force/recovery paths retained	Orders/PR Dashboard/Firefly input baselines prepared; deployment and deliberate image-refresh acceptance pending
4: Failure isolation	Parallel host-local pull/deploy chains; earlier Firefly batch; independent app children; parent failure retained	Integrated failure simulation and live acceptance pending
5: Reproducible version policy	Core/Periphery/helpers pinned 2.3.4; unused tag variable removed; exact LXC installer recipe; explicit targets intentionally retained	All four live versions verified 2.3.4; helper cached; Compose pin application pending
6: Guardrails	Recursive graph/reference/policy validation, disabled/unknown operation rejection, dependency ordering and corrected shell guard; CI regression suites	Integrated regression validation pending; no production hook invocation performed
7: Ownership/reconciliation	Explicit timezone and linked-Repo Sync bootstrap; new daily native image check replaces the disabled legacy schedule; missed-trigger deployment recovery explicitly manual	Reviewed external Sync adopted declarations and native-check schedule; no scheduled full deployment introduced

The earlier uplift contained 20 Procedures, including 7 internal/general chain Procedures, 3 bootstrap-maintenance Procedures and scoped deployment/retry/canary paths. The extension is now bootstrapped: 26 live Procedures + 2 Actions (6 Repos and 21 Stacks unchanged). The 19 automatic Stack targets in the general graph remain conditional and selected once; the three image cohorts are separate manual roots. See the operating and rollout runbook and implementation journal.

Broader shared-environment exposure and sampling work remains outside this implementation; only the new dispatch token has a separate Core-only environment. Native tag selection was evaluated but not adopted: explicit names preserve host/app boundaries and are checked recursively. New resource additions need external reviewed bootstrap Sync because the relay refuses to equate hidden resources with absent ones.

Current overlay — fleet API routing recovered; edge still quarantined¶

Recovery applied 2026-10-27 10:03:15 UTC. The 08:14 LAN hold containing orphan edge run 40469274207 also blocked fleet dispatch's Validate job because RUNNER_FLEET_API_RUNS_ON selected LAN. Fleet run 40472345328, created 08:16 UTC and canceled 09:31 UTC, had Validate queued with no runner/steps and downstream jobs skipped. This was collateral routing blockage, not proof that nothing could place: general CI Security jobs actually placed at 09:17 UTC; Hyper-V guests going offline after 900s idle is expected. Gmail Python/Maintenance jobs succeeded separately; other overall CI failures remain unfixed and outside this work. Earlier fleet run 40469102836 resumed only the exact Gmail CI runner; 40471032755 was a Gmail Docker drain dry run only, not a fleet-wide drain. Official GitHub status was all operational at inspection; that does not rule out an individual ghost run.

Only repo RUNNER_FLEET_API_RUNS_ON changed to ["self-hosted","Linux","X64","homelab","ci","homelab-ci-02"]. The prior LAN value is backed up locally in artifacts/cutover-20261026/fleet-api-routing-recovery-20261027.json. DEPLOY_RUNS_ON remains LAN; RUNNER_FLEET_MAINTENANCE_RUNS_ON remains ["self-hosted","Linux","X64","runner-management"]; no CI_RUNS_ON changes. No host/VM, restart, label, permission or credential changes accompanied recovery.
Direct proof: health run 40485555074 succeeded with scope=homelab, dry_run=true, authorize_disruptive=false. Validate 17s and Observe 22s both actually ran on homelab-ci-02; other mutation routes were skipped. The report was 7 online, 2 offline, 9 discovered, 10 expected, 3 drifts: successful read-only collection, not clean fleet-health acceptance.
Fleet operations are usable again; do not automatically rerun an unknown canceled mutation. The API selector covers validation, observation, planning, scheduled checks and guarded watchdog. It does not make every selected job read-only: watchdog apply can mutate only under its existing guarded approvals.
This temporary exact general-canary selector removes the held-LAN dependency, not the Hyper-V/scaler or single-instance dependency. It is not a new always-on or redundant architecture; preserve the override until approved reassessment.
Edge run 40469274207 remains queued with zero jobs, deployment workflow disabled, and normal Ansible timers active. The broad LAN hold is historical: at 10:49:47 UTC exact-run quarantine replaced it. GitHub verifies the runner online and non-deploying qualification successful; normal LAN jobs restored. No edge acceptance/deployment or global outage resolution follows from either recovery; the API route remains unchanged.

See the parent recovery journal and canonical routing runbook.

Current checkpoint — edge dispatch blocked before job creation¶

2026-10-27 — edge_dispatch_blocked_github_queue. After PR #517 merged, the owner approved an actual deploy-reviewed-images-edge attempt, not merely preparation. Run 40469274207, attempt 1, head 35f9a67d218f78251cedb2dd26a1e1c9edfceee1, was accepted at 07:47:46 UTC.

The workflow was briefly enabled for dispatch, then immediately disabled. A bounded 20.5-minute watch ended with the original run still queued, zero jobs and zero check runs. A 60-second re-enable did not advance it. Immediate disablement may have contributed; cause is unconfirmed. This is not evidence of a server outage or runner misconfiguration.
Cancel and force-cancel both returned HTTP 409: Cannot cancel a workflow run that has not been queued yet. In-place rerun returned HTTP 403, This workflow is already running. No retry was accepted and no new dispatch occurred. The run remains unresolved/queued, not cancelled or completed.
Verified host /opt/homelab HEAD remained 0f522f0, with no relay pending intent and no edge RunProcedure; only the old CreateProcedureUpdate was present. Sync remains at bootstrap 824c148. Edge applications and Core/Periphery had no container ID/start-time changes. Edge execution is not accepted and not a no-op: the job never started.
Historical broad containment — 08:14 UTC, superseded at 10:49:47 UTC: deployment workflow was disabled_manually; owned drain marker /var/lib/github-runner/drain/homelab-lan-01 recorded 40469274207. The idle LAN runner was stopped after GitHub busy=false and no Runner.Worker were verified; service failed/stopped, runner container absent. No job was interrupted.
Both Ansible pull timers were restored active and the owned /etc/homelab/MAINTENANCE removed. Do not strand convergence. Runner qualification, bootstrap and grants remain complete: 51 Read + 5 Execute, 26 Procedures + 2 Actions.

Current containment — 2026-10-27 10:49:47 UTC: the tested job-start hook and fail-closed systemd pre-start verifier replaced the broad drain with exact-run quarantine. The hook rejects the orphan and missing/invalid/non-homelab identity; verifier protects hash, modes, mount/launcher contract and hook configuration. Only the owned drain marker was removed after tests and backed up root-private. The LAN service is active; actual container runs as 1001:1001 plus group 10002, with correct hook/read-only mount and no Docker socket. GitHub verifies homelab-lan-01 online. Single non-deploying qualification run 40491873440 on main 54ca4da and exact LAN job 122148472245 both completed SUCCESS, verified via GitHub API, including Set up runner and Run Linux and macOS qualification checks. Normal LAN job execution is restored. Workflow remains disabled; ansible-pull-check.timer and ansible-pull-apply.timer active, MAINTENANCE, relay pending.json and drain absent. See the guard contract.

Next operator action: reconcile the edge orphan through inspection/GitHub support. Earlier browser access returned 404 despite authenticated gh API access; UI cancellation was unverified. No further cancel/rerun retry, redispatch, workflow enablement or UI Procedure/Stack bypass. Retain the guard after restoration: removal requires proof the orphan is terminal with no executable pending work and deliberate reviewed follow-up. The owner merged docs-only recovery PR #519 as 4b2cb0c; rollout holds remain.

Health limits — 2026-10-27 10:53:25 UTC: 49 running including the restored runner, 38 healthy, 11 without healthchecks, zero unhealthy/restarting; earlier DNS/TLS smoke good, not authenticated application/data acceptance. OneUptime and adapter October 27 B2 gaps are closed by explicitly owner-approved exact-snapshot copies; see the completed catch-up receipt. PR #520's local-retention fix is merged, but affected live units still lacked --retry-lock=5m at the 11:55 preflight. Normal convergence and next scheduled-run validation remain pending; no forced apply, restore/retention acceptance or broader mutation permission. Image check safely failed closed at 13/14 stacks, 32 services: Rexit latest exporter + studio versus deployed studio, native prefers deployed_services, not an auto_pull issue. Gmail Docker variable absent/proposed Docker runner unregistered is separate provisioning; the aiProject August 19 ghost predates and was not caused by this work. Normal work may resume, without mass restarts or replay of unknown canceled work. See health limitations.

Historical checkpoint — bootstrap and least-privilege grants complete¶

2026-10-27 07:16:31 UTC — bootstrapped_deployment_pending. The owner-authorized bootstrap package completed; the canonical completion record records the private receipts.

Reddit received a same-key fingerprint-only refresh, no env render/key change; all seven input --check calls passed.
One external RunSync used existing homelab-deploy scoped Execute. Exact Update 6ae04f5aa10ff54a5801dc65, accepted at 07:14 UTC, completed successfully at 07:14:52 UTC, applying reviewed main 824c1487735a5ea0b8d25a1aafab8f0362d6fa92. Resource SHA-256 2551c36fe2b846ab5123e0c52f1a6823cc36f2c6ac3bbeaa2b84c0ca8e8b1c50 was unchanged from #513; the preview was 18 creates + 13 updates, no deletes/deploys. Sync last_hash=824c148, pending resource changes/deploy targets 0, no errors.
Existing admin browser/native UI API added exactly 18 per-resource grants (15 Read + 3 Execute), leaving the original 38 unchanged. Root bot readback at 07:16:31 UTC verified 56 scopes = 51 Read + 5 Execute, unchanged non-admin identity and empty user-specific/base permissions; no admin or Server/Build creation rights were added. Live sets: 26 Procedures, 2 Actions, 6 Repos, 21 Stacks, 1 Sync.
All four operator retry webhooks are actually false. Stored inline Action code exactly matches reviewed source; Actions have startup/schedules/webhooks disabled. Native check is live at daily 03:00 America/Los_Angeles; Global Auto Update's schedule is disabled (reviewed source retains skip_auto_update=true). Backup 01:00 and key rotation 06:00 are retained.
Configuration bootstrap only: no ProcedureRun/ActionRun, workload deployment, build, workload source pull or restart. Core's normal Sync-cache fetch is not a workload pull. Host /opt/homelab remains 0f522f0; #516's runner proposal on main is not applied there. Both pull timers are active, owned maintenance marker removed, no drain/pending relay intent. Core/Periphery retain 2026-10-25T11:33 starts; workflow is disabled_manually.

Next owner action at that bootstrap capture (superseded): review the completion PR, then choose approval for the first scoped workflow deployment or optional image-publisher provisioning. First deployment/application smoke checks and unchanged retry remain pending. App/Core PAT are still unprovisioned; manual GitHub check remains available, but manual native/Action execution was not tested. Do not repeat bootstrap approval or automatically enable dispatch; no scheduled general deployment.

Historical input preparation — 2026-10-27 05:25:28 UTC¶

The owner-approved window 2026-10-27 05:24:58–05:25:28 UTC completed preparation for oneuptime-incident-adapter, paperless-stirling-bridge, firefly, orders and gh-pr-dashboard. Phase was then five_inputs_prepared_reddit_baseline_pending, bootstrap_ready=false, not global input acceptance. See the canonical preparation record.

Native host-root render, private HMAC-state publication and --check all passed (exit 0) for the five profiles. All six rendered outputs, including Orders OAuth client JSON, were byte-identical to pre-existing files. Five private state directories are 0700, with 0600 keys/fingerprints and valid 1001:1001 ownership.
Before/after credential gates passed for all five scoped profiles: SOPS, canonical configuration and actual running environments matched. Orders OAuth also matched the actual pinned mounts in both containers. Fresh password-authenticated read-only PostgreSQL 15 adapter and PostgreSQL 18 Firefly sessions passed; roles were unexpired, matching password HBA rules were verified, and HBA file mtime preceded postmaster start. No HBA reload, credential rotation, SQL writes or network token validation occurred.
Existing LiteLLM read-only check passed. Existing Reddit check failed stale before preparation and was still failed at that capture, not repaired or folded into the scoped credential pass. Both protected profiles' env/key/fingerprint bytes and metadata were unchanged.
Root read-only HMAC RCA proved the stored Reddit fingerprint exactly matches current source HEAD 3f0b0c9fd6b520fc8c12e3f73b1e0b93aaf42e46 with pre-#514 Compose; all other components were unchanged and current source was clean. Only #514's gallery-worker read-only library Compose change makes the baseline stale, not credential drift. Same-key refresh without env rendering or rotation had not yet been approved or performed at that capture; the completed reconciliation is recorded above.
Host /opt/homelab independently advanced 8bf7aa5 → 0f522f0 during preflight (#514 Reddit and #515 Phoenix retained). The 645bc3c-pinned checkout helper refused unexpected HEAD before fetch/merge; no operator host-checkout write or rollback occurred. The five profiles' input code/configs were byte-identical across 645bc3c..0f522f0; preparation used current 0f522f0, preserving four dirty tracked files.
Initial custom-auditor compatibility errors (age key root:docker 0440, legal YAML standalone age-list dash, PostgreSQL inet::text/CIDR handling) were corrected only in an ignored operator artifact. No host permissions changed; those false checker failures were not runtime failures.
Application and Core/Periphery container IDs/start times stayed unchanged through the window; no affected container was unhealthy and all healthcheck targets were healthy. Both pull timers were briefly paused under owned maintenance, then restored active and the marker removed. The runner was never drained/restarted; no relay pending intent remained. Workflow stayed disabled; no Sync, grants, deployment or Core change occurred. Private input-preparation-result-20261027.json, window receipt and before/after credential-gate evidence remain in the existing root-private bootstrap directory. This preparation preceded the separately authorized bootstrap completion above; it did not itself authorize Sync, grants or deployment.

Historical bootstrap guard — 2026-10-26 20:24 UTC¶

PR #513 merged at 20:20:07 UTC as 645bc3c5467f82ef749048ef296b457a48476e39; all 26 checks passed. Native pending cache naturally refreshed to 645bc3c at 20:24 UTC, remote-content SHA-256 2551c36fe2b846ab5123e0c52f1a6823cc36f2c6ac3bbeaa2b84c0ca8e8b1c50. The preview had 31 changes: 18 creates (16 Procedures, 2 Actions), 13 updates (6 Stacks, 3 Repos, 4 Procedures), with no deletes, deploy targets, errors, Variable or UserGroup changes. All four new retries (deploy-litellm, deploy-oneuptime-incident-adapter, deploy-paperless-stirling-bridge, deploy-firefly) had webhook_enabled=false in merged source and PENDING preview, not live-created Procedures at that capture. Sync was then unexecuted; six legacy scoped webhooks were unchanged and Sync's enabled webhook means workflow disablement is not a global lock.

At that capture, phase was runner_qualified_bootstrap_pending, host was 8bf7aa5 and normal Ansible apply was left alone. Source-only Inputs.snapshot(rendered=False) passed all six profiles without SOPS decryption/output writes, and LiteLLM/Reddit checks both exited 0; that was not DB-authentication proof or the later Reddit result. Five-profile preparation approval was then pending; the historical approval scope is superseded by the completed preparation above. The unsafe 19:25 UTC #512 c9a0fd7 preview (SHA-256 prefix a1b2d997, managed=false, delete=false) omitted four retry flags and inherited v2.3.4's true default; it is not approval to execute. Guard validation recorded 93 focused and 275 combined tests, resource validation and inline Action agreement. At that capture, live bot access was 36 Read + 2 Execute, with no user-specific/base permissions, admin or Server/Build creation rights; no grants changed during that guard review.

Credential-access follow-up — scoped acceptance complete, 2026-10-26 UTC¶

The post-merge checkpoint records deployment run 40299611768 failing at credential loading before host fast-forward, ResourceSync, Procedure execution or new pending intent. At that capture, the legacy file /etc/homelab/komodo-deploy.env was correctly 1001:1001 0600. The actual LAN container ran as 1001:1001 plus GID 10002, not host Docker GID 987. During active Ansible apply, /etc/homelab changed from inaccessible root:docker 0750 to readable root:root 0755. The common ansible-pull/monitoring-deadman and operator-tools/labctl roles disagree about that shared parent; a one-time successful read concealed it.

The follow-up removes the Komodo dependency on that parent rather than changing the shared roles or adding Docker membership. Those policies and labctl traversal remain an unresolved separate issue; /etc/homelab:ro is retained for labctl. The new source contract is:

API/default file /etc/komodo-deploy/komodo-deploy.env, 1001:1001 0600, with root:1001 0710 parent: group search-only, no listing/write. Renderer validation covers root ownership, exact group/mode, no extended/default ACL and no ancestor symlinks even for --check, validation and unchanged renders; unsafe directories are rejected, not repaired.
Enabled LAN preflight and actual runner use the same-path read-only directory --mount. Missing source fails; a directory bind sees atomic replacement, unlike a file bind that pins the old inode. No new Docker socket or supplementary Docker group is introduced.
Root-only explicit --migrate-legacy verifies the existing legacy pair equals reviewed SOPS before creating/rendering the new path. A different new credential or unsafe existing directory is refused. The old file, ciphertext and age identity remain untouched; no rotation or recapture. CLI status is 0 unchanged/valid, 3 changed/would change, 1 failure; --check makes no filesystem mutations.

The prior approved-cutover checkpoint (2026-10-26 UTC) recorded /opt/homelab fast-forwarded to 9240794, retaining #509. The existing credential was migrated unchanged to the dedicated path with the metadata above verified; the legacy file was retained. Temporary runner-identity read-only preflight passed, not actual recreated-runner acceptance. Fresh recheck found the deployment workflow disabled, homelab-lan-01 offline with busy=false, its container absent and drain marker present, and no relay pending intent. No ResourceSync, deployment or Core restart was performed.

At that checkpoint, Ansible timers were active and the owned maintenance marker was removed. Apply finished successfully at 10:51 UTC; ansible-pull-check recorded an older exit 2 from 05:48 UTC, before cutover. These are distinct observations, not a claim that all convergence checks passed.

The subsequent waiting checkpoint recorded the 11:22 UTC normal apply still active after a 180-second wait (exit 75: waiting, not failure), without a new maintenance window or runner start. That is historical, not the current operating state.

Historical activation checkpoint — 2026-10-26 12:06 UTC, runner_activated_pending_convergence: the owner merged PR #510 at 2026-10-26 11:27:39 UTC as 0416d8e; all 26 PR-head checks succeeded. Review and controlled activation are complete. Normal apply finished 12:01:20 UTC, exit 0, without interruption; the 11:49 UTC check-service exit 0 may be a lock skip, not full convergence proof. The locked host fast-forward to 0416d8e preserved local edits/paths. Credential-only common and allowlisted runner-only applies succeeded; runner 24873 was online/idle. Actual default-identity, directory-mount, SOPS consistency and read-only API checks passed, but only /etc/homelab as root:root 0755 was observed at that checkpoint. Both-state and natural-convergence acceptance were then open. See the historical activation record.

Completed acceptance — runner_qualified_bootstrap_pending, 19:05 UTC: source, main and host were 0416d8e at that capture, not the later bootstrap preflight. The bounded read-only observer watched the next natural apply 18:54:43–19:04:57 UTC, exit 0, full recap ok=439 changed=29 skipped=98 failed=0. The same actual container passed credential, SOPS, default-identity/mount and read-only API probes before, during the naturally restrictive root:docker 0750 state and after return to root:root 0755. Credential/both-mode/natural-convergence acceptance is complete, without chmod, forced apply, timer pause, markers, restart or credential changes. The separate 17:50–17:51:58 UTC check-mode run failed (exit 2) at operator-shell : Set operator shell home with an undefined value (ok=135 changed=3 failed=1 skipped=76); it is not fixed and this is not whole-host check-mode acceptance. See runner evidence and the canonical acceptance record.

The reviewed flags remain infra-services-only, with role defaults false. At the 2026-10-26 acceptance capture, both timers were active; maintenance/drain markers and relay pending intent were absent. Deployment stayed disabled; no ResourceSync, grant change, deployment or Core/Periphery restart occurred in that window. Preserve unrelated host files and relay lock/pending recovery state.

Consumer rollback keeps the new mount and explicitly sets the existing KOMODO_DEPLOY_ENV_FILE=/etc/komodo-deploy/komodo-deploy.env. Restoring old mounts restores the bug; keep dispatch paused. Legacy cleanup requires separate explicit approval, never automatic migration cleanup. The rollout and bootstrap procedure remains authoritative. External Sync/grants subsequently completed under separate approval; optional image-publisher App/Core PAT and workload deployment remain gated. Runner acceptance alone authorizes none of them.

Image-update extension — source merged, bootstrap complete 2026-10-27¶

The extension was source-only on 2026-10-26; its resources, schedules and relay grants are now live as recorded above. Optional credentials and execution validation remain pending. The initial runtime table below remains dated history. The image-update runbook is the operating contract.

Daily 03:00 America/Los_Angeles now belongs to check-image-updates, invoking its inline Action for 14 fixed native Stack checks. It checks same-tag digests only; pinned/local-build images are explicitly unsupported and incomplete accounting fails. Native false is not detected or unknown, never proof of current versions. Legacy Global Auto Update remains manual, schedule_enabled = false, with skip_auto_update = true. No Procedure enables auto_updates = true.
Manual check-image-versions and confirmed propose-image-pin-updates invoke request-image-pin-updates, dispatching GitHub check/propose. Weekly Dependabot is unchanged. The bounded registry policy covers 13 image fields in six stacks, across edge (Traefik/Homepage/AdGuard), documents (Paperless/Stirling) and finance (Firefly). Same-minor patches preserve prefix/suffix; major aliases remain same-tag digest-only with no semantic patch guarantee. OCI index/manifest digests are verified. Errors/private registries/local builds/digest-only/unsupported/truncated results fail closed; they cannot become false up-to-date reports or partial proposals.
The writer can change only allowlisted Compose image fields, literal Quick Reference Image table refs and future proposal history. Arbitrary documentation is never rewritten; warnings need human review. No automatic Rexit, OneUptime, Komodo or local-build/source pin updates. Native Rexit metadata checking is a separate, safe-unknown-capable path, not private-registry version-writing authority.
Publishing creates a draft PR using a dedicated repository-only GitHub App (Contents:write + Pull requests:write), with no merge, deployment, force operation or GITHUB_TOKEN fallback. Required vars.IMAGE_UPDATES_APP_CLIENT_ID and secrets.IMAGE_UPDATES_APP_PRIVATE_KEY are not provisioned. App-created PRs must trigger normal CI; the App's permissions are broader than the code's path allowlist.
The .ts bodies are generated into resources.toml inline by scripts/render-komodo-actions.py; --check verifies agreement. Core has no code-checkout mount and must never execute runner-writable file paths. The optional Core-only env is /etc/homelab/komodo-core/image-updates.env (root-owned parent 0700/file 0600), rendered by scripts/render-komodo-image-update-env.py from future SOPS ciphertext. Only an example exists. The fine-grained dispatch PAT needs Actions:write on this repo, not Contents:write; token authority exceeds one workflow, constrained by a fixed code whitelist. No new token was created. Controlled Core recreation is required to load it; manual GitHub check needs no Core token or App.
Correlation uses workflow input request_id and exact run name Komodo image updates / op / cohort / uuid. The Action generates a UUID, dispatches once and polls the exact run for at most 35 minutes. Unknown outcomes require reconciliation, never blind retry or a latest-run success inference.
The existing result-aware workflow permits four roots: default deploy-infra and deploy-reviewed-images-{edge,documents,finance}. Cohorts explicitly pull homelab then conditionally reconcile their exact whole changed stacks from current main, including non-image changes. Review all relevant diffs and drain independent writers; do not bypass the workflow through direct UI execution.

The 2026-10-26 baseline was 38 grants (36 Read + 2 Execute) over ten live Procedures and no Actions. Bootstrap added sixteen Procedures and two Actions; 2026-10-27 07:16:31 UTC bot readback verified 56 effective grants = 51 Read + 5 Execute over 26 Procedures, 6 Repos, 21 Stacks, 2 Actions and 1 Sync. Exactly 18 per-resource grants were added; the original 38 stayed unchanged. Execute covers only Sync infra-services-01, deploy-infra and deploy-reviewed-images-{edge,documents,finance}; all others are Read. Operator check/propose Execute is not required by the bot. Native administrative delegation still makes reviewed Git/Sync/Procedure/Action definitions a trust boundary.

Earlier public-registry smoke checks for Traefik, Homepage and Stirling were metadata only, without image pulls, pin writes or PR publishing. On 2026-10-26 UTC the manual GitHub edge check run 40299854748 succeeded on merged main (UUID 3a3c855b-381b-4787-b625-3c6fb6ec22ab). It reported Traefik v3.7 → v3.7.16 and initial digest-pin proposals for unchanged Homepage v1.13.2, AdGuard v0.107.79 and Unbound 1.22.0; publish was skipped and the changed-file list was empty. No pins, PR, image layers, Sync or deployment resulted. This is not all-cohort or Komodo Action-dispatch acceptance. PR #506 integration/CI and the later #508 migration checkpoint are recorded in the journal. #510 review, controlled runner activation and scoped credential/both-mode/natural-convergence acceptance are complete, as are the subsequent 2026-10-27 bootstrap/grants. Optional credentials/Core maintenance, native/Action execution validation, proposal publishing and deployment/application acceptance remain pending.

Evidence and scope¶

Original audit working branch: feat/nocturne-result-filters, starting at 4e7a8d1.
Production checkout and remote-main snapshot at the initial audit: fec3dbc720289c7371eeb6654b1f6f4dc9817241. That production snapshot was ahead of the original working branch. Findings below were reconciled against it and selected live database fields, not inferred from older local declarations alone.
Later packaging baseline: remote main 7e7f79b6, with manual-only dispatch and newer operational corrections. The fresh isolated clone preserves those contracts; the original dirty working tree remains untouched. See the packaging record.
Capture: October 25 in America/Los_Angeles (October 26 UTC).
Inspected container metadata, selected resource/schedule fields, and execution outcomes on infra-services. Database authentication stayed inside the container; secret values and deployment logs were not exported.
Compared behavior with upstream v2.3.4 source. LXC versions were not measured during the initial capture; a later read-only check confirmed both at 2.3.4 (see version evidence). Image-update availability and long-term load were not measured.

Observed runtime¶

This table is the initial fec3dbc capture, before source uplift and guided preparation; it is not a fresh observation of the packaging checkout.

Area	Observation
Core / infra-services Periphery	Both v2.3.4; Compose still uses floating `:2` tags
Core image digest	`sha256:91b2afd7b3bb2abea0ebd3046d65ec457eb1f3d505868d1d60cb0f8562490ef9`
Periphery image digest	`sha256:fb4b52961fcce7a31e3776930e422967b6640f653de876f59f26dea243e29beb`
Resources	3 Servers, 6 Repos, 21 Stacks, 10 live Procedures, 1 ResourceSync, 0 Actions, 0 Builds, 1 Builder
Git-declared policy	19 `auto-deploy`, 1 `self-managed` (Komodo), 1 `manual-upgrade` (OneUptime); 7 declared Procedures
Native tracked dependencies	Reddit Explorer tracks `.deploy-inputs/fingerprint` with `requires = "Redeploy"`; other 20 Stacks have empty `config_files`
Image automation	All 21 Stacks have `poll_for_updates = false` and `auto_update = false`
Sync	`infra-services-01` links to Repo `homelab`; resource path `services/komodo/resources.toml`; `managed = false`, `delete = false`
Deployment schedule	`deploy-infra` has an empty schedule; no Actions exist
Bootstrap schedules	Database backup daily 01:00, Global Auto Update 03:00, server-key rotation 06:00; enabled, inheriting Core timezone
Sample memory	Core ~90 MiB, Periphery ~106 MiB, Mongo ~158 MiB; one `docker stats --no-stream` sample, not a capacity study
Recent outcomes	Five sampled `deploy-infra` Updates completed successfully in ~38–256 seconds; corresponding Syncs took ~1.4–11.7 seconds

At the initial capture the three bootstrap Procedures were live but absent from that revision's resources.toml. Their presence is not inherently drift: initialization resources are enabled. However, their ownership, restore behavior and schedules deserve explicit documentation. Global Auto Update is not a Git reconciliation fallback, and the observed Stack flags do not permit it to auto-update those stacks.

Probe limitation¶

periphery --version reports its version normally. Do not use core --version as a read-only version probe: this build starts Core instead. The probe printed v2.3.4, attempted startup and exited because port 9120 was already occupied. Subsequent container metadata showed the original Core and Periphery still running with unchanged start times and zero restarts. No deployment was requested; startup-side database activity from that short-lived process was not independently audited. Future checks should use existing startup/version metadata rather than launch another Core.

What is already working well¶

Git-authoritative ResourceSync, file-on-host stacks, and shared checkouts avoid duplicating the monorepo per stack and preserve Ansible/SOPS ownership of secrets.
Komodo manages three Docker hosts without taking ownership of Saltbox containers or inventory-only customer applications.
Native OIDC, PKI, explicit deployment policy and a self-managed control plane are appropriate boundaries. Do not make Komodo restart itself through deploy-infra.
OneUptime's explicit maintenance/migration gate is intentional, not an automation gap.
Reddit Explorer already demonstrates the right efficiency pattern in production: scoped desired-input fingerprint, native config_files, DeployStackIfChanged, and a separate unconditional retry Procedure. Its source pull now runs after the ordinary infrastructure batch, not before it.
The captured fec3dbc workflow excluded **/*.md, reviews/** and root tests/** from push triggers, but mixed runtime changes could still trigger it. This is historical: newer main 7e7f79b6 is manual dispatch only, preserved by the uplift; no push exclusions are needed for the current policy.
The existing validator and Reddit mutation tests are useful guardrails. Extend them; do not replace them with a generic schema-only check.

Execution model at the audit capture¶

flowchart TD
    A[Eligible merge to main] --> B[LAN Actions runner and policy checks]
    B --> C[Safe host checkout fast-forward]
    C --> D[Request ResourceSync]
    D --> E[Fixed settling delay - default 15 seconds]
    E --> F[Request deploy-infra]
    F --> G[Pull three homelab checkouts]
    G --> H[Conditional batch - 15 stacks]
    H --> I[Pull Reddit source and publish fingerprint]
    I --> J[Deploy Reddit only if inputs changed]
    J --> K[Pull and unconditionally deploy Orders]
    K --> L[Unconditionally deploy Firefly]
    L --> M[Pull and unconditionally deploy PR Dashboard]
    N[Five-minute resource refresh] --> O[Refresh cached state and pending changes]

The five-minute refresh does not execute Sync, pull the host checkout, or run deploy-infra. There is no observed autonomous Git-deployment fallback schedule. HTTP 200 from either webhook acknowledges the request, not successful completion. The 15-second delay is not a dependency barrier. The sampled successful Syncs fit within it, but that does not establish a guarantee.

Stages run sequentially; executions within a stage, including expanded batch executions, run concurrently. A failure stops subsequent stages. A long allowlist is therefore neither a serial deployment queue nor a host-isolation boundary.

Original prioritized findings and acceptance criteria¶

1. High: cover real deployment inputs¶

Gap: litellm, oneuptime-incident-adapter and paperless-stirling-bridge are in the conditional batch and build local application content, but lack tracked build-input dependencies. Code, Dockerfile, dependency, hook, and configuration-only changes can therefore be missed by Compose-file change detection. Other stacks also need an input inventory; changing a bind-mounted file is not necessarily enough for its application to reload it.

Use native capabilities: add config_files with an explicit requires action for suitable non-secret files. Use the existing Reddit fingerprint pattern for aggregated source/build inputs or secret-bearing runtime inputs. Files belong in config_files, not Compose file_paths.

In v2.3.4, tracked paths are literal UTF-8 files, not globs or recursively hashed directories. The default action is None; explicitly choose Redeploy or Restart. A restart-only action does not replace a deployment that must run pre_deploy.

Render inputs and publish the desired-input fingerprint before conditional execution. A skipped deployment never runs pre_deploy, and tracked files are read before that hook. Use git rev-parse HEAD for external checkout revision markers; .git/HEAD normally contains an unchanged symbolic reference.

Do not track plaintext .env/compose.env: tracked content enters Komodo's cache/display/diff machinery. Reddit's existing implementation uses keyed HMAC, private state and atomic output, and verifies freshness in predeploy. Reuse those properties rather than creating ad hoc unkeyed secret hashes. A source commit alone also requires a clean/build-context policy for untracked inputs.

Acceptance: source-only, dependency-only, config-only and secret-only changes select the intended stack; unrelated changes and a second unchanged run do not; missing/stale inputs fail closed; no plaintext secrets appear in Komodo diffs.

2. High: make the relay wait for results, not elapsed time¶

Gap at the initial fec3dbc capture: scripts/trigger-komodo-deploy.py sent signed Sync and Procedure webhooks with a fixed sleep between them and never checked the terminal Update. That revision's if: always() summary claimed Sync completion even on failures. Newer main corrected summary/dispatch policy; the uplift preserves those corrections and replaces the remaining acknowledgement-only relay. Workflow concurrency alone cannot cover a Procedure still running after the relay exits.

Recommendation: use Komodo's native API with a narrowly authorized automation identity to execute RunSync, capture that execution's Update ID, and wait for status = "Complete" and success = true before requesting RunProcedure. Wait for that exact Procedure Update too. Fail on failure or a bounded deadline; timeout means outcome unknown, not cancellation. Do not poll a generic "latest" Update that might belong to a different operator.

The v2.3.4 raw response identifies persisted Updates as _id.$oid; GetUpdate reads them. Single RunSync and RunProcedure executions provide correlation. Do not substitute a direct batch/if-changed call and assume it has the same parent Update contract. Retain Sync outside the Procedure whose definition it manages.

This needs a SOPS-managed API credential and permission review; do not widen the runner with a Docker socket or expose Core publicly. Until implemented, the runbook's manual terminal-Update verification remains required.

Acceptance: a Sync exceeding 15 seconds delays deployment; a failed Sync prevents it; failed/unfinished deploys cannot produce a success summary; overlapping dispatches are handled deliberately; no secrets enter job output.

3. Medium: extend selective application deployment¶

Gap: Orders and PR Dashboard still receive unconditional DeployStack calls on general deploys. Orders explicitly builds with --pull and forces container recreation. Reddit is no longer an example of this problem. Firefly's unconditional execution is a deliberate environment-render workaround rather than an unconditional build.

Recommendation: adapt Reddit's desired-input contract to Orders and PR Dashboard. First reduce when Orders executes; preserve its required build/recreate behavior until separately proven unnecessary. Give base-image refresh an explicit policy rather than relying on unrelated merges to trigger --pull.

For Firefly, track the encrypted/rendering inputs and preserve render-before-up ordering before removing its unconditional refresh. Keep force-deploy/recovery paths. Do not call a desired-input fingerprint a successful-release marker.

Acceptance: unrelated merges do not rebuild/recreate these applications; actual source/config/environment changes do; force retry and deliberate image refresh work.

4. Medium: isolate independent failure domains¶

Gap: failure to pull either LXC checkout still stops all later infrastructure deployment. Reddit or Orders failure can still prevent Firefly's environment refresh. Moving Reddit behind the ordinary batch fixed one dependency problem, not all of them.

Recommendation: move Firefly ahead of fragile application work, then consider host-scoped or application-scoped Procedures coordinated as independent chains. Keep pull-before-deploy and fail-closed behavior within each chain. Do not simply ignore failures or deploy an unavailable host's old checkout. Stage smaller batches if measurements show resource contention; do not parallelize every build by default.

Acceptance: failed SonarQube/Phoenix preparation does not block eligible infra-services work, and failed application builds do not suppress Firefly; no affected host deploys stale inputs.

5. Medium: use the installed version's capabilities deliberately¶

Gap: floating :2 tags obscure upgrade/rollback expectations, while the explicit batch allowlist is still explained as necessary because tags do not work.

Observed: the infra-services pair is v2.3.4. Upstream introduced batch tags in v2.3.0 and fixed Procedure batch-tag filtering in v2.3.3. The old blanket statement is obsolete for this pair. Keeping explicit target lists remains a valid safety choice.

Recommendation: record/test matching exact versions or digests, including LXC Peripheries. Then optionally replace duplicated allowlists with tested native tag selection, preserving the three disjoint target sets: ordinary conditional batch, special conditional application, and unconditional exceptions. Matching all auto-deploy stacks indiscriminately would lose that distinction or deploy twice. Do not remove homelab policy validation or enable automatic image deployment merely to increase feature usage. COMPOSE_KOMODO_IMAGE_TAG currently does not control the literal image tags in compose.yml.

Acceptance: reproducible upgrades/rollback and a target-selection test proving Komodo/OneUptime exclusions and exactly-once selection of all eligible stacks.

6. Medium: strengthen remaining validation and hook failure handling¶

Production's Reddit-specific checks are stronger than this older working branch's. Nevertheless, in-memory production policy probes still accepted a disabled ordinary batch stage, a nonexistent initial PullRepo target, and an extra BatchDeployStack execution targeting Komodo. These are validator holes, not current live actions.

Require valid Server/Repo references, enabled required stages/executions, and a reviewed set of execution types. Preserve existing mutation tests and add negative fixtures for these cases. An upstream resource schema can supplement, not replace, policy checks.

The main homelab on_pull hook also joins into chown ... && test ... || chmod .... A successful chmod can mask failed ownership repair. Group the guard as chown ... && (test ... || chmod ...) and behavior-test both failures. The Reddit hook already has stronger sequencing tests; this is not a reason to rewrite it.

Acceptance: each mutation is rejected; ownership or permission-repair failures remain failures. No production hooks were changed during this assessment.

7. Medium: make reconciliation and bootstrap ownership explicit¶

Document the three live bootstrap Procedures and linked-Repo Sync configuration alongside Git-managed resources. Keep Sync managed/write-back and deletion off unless ownership changes are explicitly reviewed.

After conditional input coverage and result-aware orchestration are established, decide whether a scheduled missed-trigger reconciliation is worth adding. Native Procedure/Action schedules exist, but do not schedule the current full pipeline every five minutes: it still contains unconditional work and a self-update boundary. A reconciler must perform external Sync → successful completion → deployment without syncing its own running definition, respect maintenance/upgrade gates, and avoid duplicate concurrent runs. Its schedule, identity and recovery path must be declared.

Acceptance: bootstrap from documented inputs reproduces the intended schedules; a missed trigger is either explicitly manual recovery or handled by a tested bounded reconciler, never assumed to be repaired by polling.

8. Lower: narrow environment exposure; tune sampling only with evidence¶

Both Core and Periphery receive the entire compose.env, including Core-only settings and credentials. Separate service-specific environment inputs or use native file-backed secrets where appropriate, retaining Ansible/SOPS as owner. Review terminal access and automation permissions. Periphery already has the Docker socket, age key and writable host paths; it is a privileged control plane, not a strong sandbox.

KOMODO_UI_WRITE_DISABLED disables UI editing; it is not an API write-denial boundary. API authorization and sync ownership still matter.

The sampled ~354 MiB total memory does not justify replacing MongoDB or aggressively tuning monitoring intervals. Measure busy-period CPU, retained statistics and disk usage first. Core resource refresh, Core monitoring, Periphery sampling and image update polling are different mechanisms. Keep useful operator diagnostics without trying to duplicate the Prometheus/Grafana platform.

Retry semantics to preserve¶

For v2.3.4, a failed predeploy/build/Compose-up attempt does not generally advance the deployed-content baseline. Successful Compose up followed by failed postdeploy can advance it despite the overall failed Update. A subsequent unchanged conditional run may skip that failed verification. Preserve targeted force-deploy or verification retry paths and record application acceptance separately from Compose success.

The old blanket comment that any failed conditional attempt records a deployment hash is too broad for this version. Existing dedicated retry Procedures remain useful.

Documentation and ownership contract¶

Future implementation must update, together:

Resource/Compose/relay configuration and focused regression tests.
The Komodo service README and explicit-deployment runbook.
Affected application READMEs, input/secret ordering, retry and rollback instructions.
Agent conventions, journal and navigation; inventory/generators if ownership or host placement actually changes.
This audit's statuses with separate implemented, deployed and verified evidence. A green webhook job or source diff is not rollout acceptance.

Keep PLAN.md as the historical master specification; use the live operating references for as-built behavior. Do not rewrite old journal evidence as though these improvements were present then. The July reliability audit now links here for corrections to polling, Sync webhooks, UI/API permissions and self-sync advice.

Sources¶

Repository evidence below is pinned to the inspected production revision:

Upstream evidence: