Orders release identity guard — 2026-10-28¶

The Orders-only guard binds a deployment to fresh source identity and verifies copied runtime bytes before the web or bot application starts. The guard is active; Orders-only activation was accepted at 2026-10-28 03:08 UTC. Earlier separately authorized rollout left application PR #20 healthy at 3fdf3f2; that is a baseline observation, not acceptance of this new guard.

Status and scope¶

Area	Status
Earlier live inspection	Build, build-pull, and recreate enabled; mutable image tag and no post-deploy hook
New guard	Active using homelab `3a02ec3`; scoped Sync and native `deploy-orders` acceptance recorded below
Validation	Initial 244 targeted tests and disposable Docker smoke passed; CI follow-ups are recorded below
Rollout	Owner-approved Orders-only activation completed; this does not clear unrelated general/edge deployment gates
Other application profiles	Selection logic unchanged; the shared helper's new hash invalidates old fingerprints and needs normal refresh before a later deploy

The service's release and deployment contract is the operator reference. This entry distinguishes implementation, validation, and activation; it is not itself authorization to deploy.

Guarded workflow¶

flowchart TD
    A[Managed Orders source pull] --> B[Prepare release from clean fresh main]
    B --> C[Render secrets and literal revision then check release]
    C --> D[Refresh input fingerprint]
    D --> E[Conditional deploy or explicit force retry]
    E --> F[Pre-hook checks fingerprint and fresh release]
    F --> G[Build revision image with base pull]
    G --> H[Recreate web and bot without build or pull]
    H --> I[Startup verifies bytes before application exec]
    I --> J[Post-hook proves health image identity and runtime bytes]

release.py prepare runs before secret rendering; release.py check runs after rendering. Both require clean on-main source, including ignored/untracked contents, no hidden index bits, and exact committed bytes and executable modes. HEAD must equal local origin/main and a fresh git ls-remote origin refs/heads/main result. Network, authentication, source movement, or metadata failure refuses the operation: there is no cached fallback or automatic Git repair.

The renderer emits exactly one literal ORDERS_SOURCE_REVISION=<sha> assignment using the full commit SHA. Compose requires homelab/orders:${ORDERS_SOURCE_REVISION:?Orders release revision is required}; web and bot share that image and its build-time org.opencontainers.image.revision OCI label. Both use PYTHONDONTWRITEBYTECODE=1 and the mandatory entrypoint python -B /run/orders/verify-release.py.

The public .deploy-inputs/orders-release.json manifest and readable .deploy-inputs/verify-release.py copy are 0644. Their directory stays 0700 because it also contains the sensitive existing HMAC key. Only these two files are bound read-only, as /run/orders/release.json and /run/orders/verify-release.py; never bind the whole directory or expose the key. Both generated artifacts join the existing input fingerprint. An Orders source docs-only commit changes the full revision and therefore the fingerprint/redeployment decision; other profiles retain their existing behavior.

Komodo keeps run_build = true and build_extra_args = ["--pull"]; up uses extra_args = ["--force-recreate", "--no-build", "--pull", "never"]. Predeploy checks fingerprint plus fresh release. Postdeploy runs bash scripts/release-check.sh post, proving both containers healthy on the same image ID, expected tag, and OCI revision label, then hashing their actual copied runtime files with the trusted current verifier rather than trusting labels or an older mounted script. The host wrapper uses native Python/Git or the existing already-local pinned 2.3.4 nsenter bridge; it never pulls a helper image. Neither check nor post mode renders secrets.

Startup rejects a missing/malformed manifest or mismatched, missing, or extra copied runtime files before application exec, hence before database, Gmail, or bot work. Retagging/relabeling an old image with different runtime bytes cannot satisfy that proof. The byte contract covers app/ and requirements.txt, not every OS or installed dependency file.

Validation¶

Validation ran from the isolated checkout at homelab base 68fbae0, using the existing repository-root virtual environment. Tests use synthetic files/Git repositories and mocked provider commands, not production secrets or data.

Command	Result
`../../.venv/bin/python -B -m unittest discover -s tests -p 'test_orders_*.py'`	73 passed
`../../.venv/bin/python -B -m unittest discover -s tests -p test_komodo_deploy_inputs.py`	40 passed
`../../.venv/bin/python -B -m unittest discover -s tests -p test_komodo_deployment_graph.py`	43 passed
`../../.venv/bin/python -B -m unittest discover -s tests -p test_trigger_komodo_deploy.py`	88 passed
`../../.venv/bin/python -B tests/orders_release_container_smoke.py`	Passed: real Compose rejects missing revision; verified non-root web/bot fixtures start; old bytes with the expected revision label do not start
`../../.venv/bin/python -B scripts/validate-komodo-resources.py`	Passed: 21 stacks, 6 repos, 26 procedures
`bash -n services/orders/scripts/render-env.sh services/orders/scripts/predeploy-check.sh services/orders/scripts/release-check.sh`	Passed
Scoped `markdownlint-cli2 --fix` then check on the service README and both journal files	Passed
`uvx --from yamllint==1.37.1 yamllint -c .yamllint.yaml services/orders/compose.yml .github/workflows/lint.yml mkdocs.yml`	Passed
`uvx --from actionlint-py==1.7.12.24 actionlint .github/workflows/lint.yml`	Passed
`../../.venv/bin/python -B scripts/check-docs.py`	Passed after adding the journal to site navigation
`git --no-pager diff --check`	Passed

The Komodo CI job now discovers test_orders_*.py, so source, startup, and post-deploy regressions run in that existing check without changing its name, permissions, or runner routing. The journal is included in site navigation.

The Docker smoke used an already-local Python 3.13 slim base by digest, synthetic images, UID 10001, network-disabled containers, and no production mounts. It did not run the application bot or spend provider quota. Release-helper tests also passed under Python 3.12.9, and the helpers parse using Python 3.11 grammar; a Python 3.11 runtime was unavailable. The existing virtual environment lacks yamllint; a pinned isolated uvx tool is used for the scoped YAML check.

Regression-first runs reproduced the mutable-tag/missing-post-hook contract, unfingerprinted generated-artifact changes, late source/metadata changes, and invalid release state. Integration tests caught a web-container replacement during the bot proof and Bash 3.2 empty-array expansion; both were fixed before the final passing run. The test expectation now requires final identity checks of both container names after the last source proof. Final read-only review found no additional material issues within the stated contract.

These initial checks do not replace current-head CI, live ResourceSync acceptance, or a controlled production deployment. The full unrelated homelab suite was not run during initial validation.

CI follow-up: isolate runner filesystem defaults¶

The first PR #522 CI run failed both the Komodo contract job and Python unit job with 25 failures and one error in test_orders_release.py; ci-gate reflected those failures. Valid synthetic fixtures inherited /runner/tmp, whose writable ancestor was correctly rejected by the production permission guard. Local macOS temporary-directory defaults had hidden that assumption.

A disposable non-root Linux container with TMPDIR=/runner/tmp, writable runner ancestors, and umask 0002 reproduced the same 25 failures/one error. The first test-only correction (fa381b0) used private fixtures under system /tmp, a scoped 0077 umask with cleanup even on setup failure, and retained LD_LIBRARY_PATH for setup-python subprocess probes. That reproduction passed, but the next CI run still failed: the runner launcher also mounts /tmp as a runner-owned tmpfs, not a trusted root-owned temporary ancestor.

Adding the actual launcher's tmpfs flags to the reproduction exposed that missing detail (26 failures/one error). The completed correction uses resolved /var/tmp for valid fixtures and keeps the scoped umask/library-path handling. No production path checks, runner permissions, workflow gates, or skips were weakened. Regressions verify caller settings are restored, setup failures clean up, and runner-owned sticky directories remain rejected by the production guard.

The branch incorporates current Homelab main through merge commit 75600bd (including rules PR #523), without rewriting its published history.

Follow-up validation	Result
New fixture-isolation regressions before the fix	1 failed, 1 passed; unsafe runner temp inheritance reproduced
`../../.venv/bin/python -B -m unittest discover -s tests -p 'test_orders_*.py'`	75 passed
`../../.venv/bin/python -B -m unittest discover -s tests -p test_ws8_operator_contracts.py`	27 passed after merging main
`docker run --rm --network none orders-ci-repro:pr522-baseline`	Reproduced 25 failures, 1 error in the 38 original release tests
`docker run --rm --network none -e LD_LIBRARY_PATH=/usr/local/lib orders-ci-repro:pr522-fixed`	First reproduction: 40 release tests passed; it did not yet model the runner-owned `/tmp` mount
Same first-fix image with the runner-owned `/tmp` tmpfs	Reproduced the second CI failure: 26 failures, 1 error
Completed fixture fix with the actual runner tmpfs/capability flags, UID 10001 and again UID 1001	41 release tests passed in each offline container
Full `../../.venv/bin/python -B -m unittest discover -s tests -q`, launched with isolated HOME/Git/GH settings and a narrow environment	Timed out at the 330-second bound; no completed full-suite result claimed

All disposable images used the same digest-pinned, already-local Python 3.13 base; Git was installed only in the test image. Test execution was offline, with no production mounts. The completed fix retained the deliberately unsafe runner temp directory, runner-owned /tmp, and permissive outer umask. The exact final reproduction command was:

docker run --rm --network none --user 10001:10001 \
  --cap-drop ALL --security-opt no-new-privileges \
  --tmpfs /tmp:rw,noexec,nosuid,nodev,uid=10001,gid=10001,size=512m \
  -e LD_LIBRARY_PATH=/usr/local/lib orders-ci-repro:pr522-runner-tmpfs

It was repeated with UID:GID 1001:1001 and matching tmpfs ownership. These flags come from the existing runner launcher; that launcher was not edited. Read-only review found no material issue in fixture isolation/restoration. Current-head CI must still confirm the complete configured suites; no live guard activation is implied.

Post-merge follow-up: read-only tests and access times¶

PR #522 finished with 27 successful checks on 55c5863, then merged as c3e6fda. The authorized activation preflight found that the main-branch Python job had failed two assertions in the 2,345-test run (47 skipped): test_check_missing_verifier_copy_is_read_only and test_public_verifier_requires_service_owner. Their saved os.stat_result values differed only in access time; reads can legitimately update that field. The Komodo job passed, but the full-suite and aggregate gate failures blocked activation. No host configuration, running container, database, or credential was changed by this preflight.

The follow-up aligns read-only test snapshots with the guard's existing mutation stamp: bytes, device/inode, permissions, owner/group, link count, size, and nanosecond modification/change timestamps. Access time is excluded; content and mutation detection are not. It updates the other same-pattern assertions in the release tests too, rather than waiting for another timing-dependent failure. Production guard code, filesystem permissions, and runner configuration are unchanged.

Validation	Result
Deterministic snapshot regression before the fix	Access-time-only comparison failed; the two content/mutation cases passed
`../../.venv/bin/python -B -m unittest discover -s tests -p 'test_orders_*.py'`	79 passed
Updated disposable Linux reproduction with runner-owned `/tmp`, UID 10001, dropped capabilities, no-new-privileges and offline execution	44 release tests passed

New deterministic tests prove that access-time-only changes are harmless while changed bytes and each of the nine mutation/identity fields remain observable. The prior local full-suite timeout is not recast as a pass; current-head CI must validate this follow-up before merge and a green main is required before resuming the already-authorized guard activation.

Activation accepted — 2026-10-28¶

The owner separately authorized activation and granted the existing homelab-deploy service user Execute on Procedure deploy-orders. Its existing Sync execution permission was used with the native filters resource_type = "Stack", resources = ["orders"]; no admin/global Write grant, new credential, general deployment, or other-stack deployment was required.

Both main-branch revisions were green before mutation: 27 Homelab checks at 3a02ec38ce4e93b72a0ea21afb4192db10aa548e and 8 application checks at d7d111f139f50b0e19685abb9828f4c01ba31bbd. The earlier CI and permission blockers were cleared; the historical sections above retain their evidence.

The bounded operator run held the shared relay lock, allowed the active Ansible apply to finish, and used an owned maintenance marker to pause later convergence. The host checkouts fast-forwarded without stash/reset/clean; incoming paths did not overlap the existing local changes, and before/after dirty-state signatures matched. Unrelated tracked and untracked work was retained.

The configured pinned helper tag was missing. The running Periphery binary was verified as 2.3.4, with image/RepoDigest sha256:fb4b52961fcce7a31e3776930e422967b6640f653de876f59f26dea243e29beb. That same already-running image was tagged locally as ghcr.io/moghtech/komodo-periphery:2.3.4; Core and Periphery were not restarted or upgraded, and the wrappers still use --pull never.

Orders inputs were rendered and verified without changing existing credential contents. A real candidate image passed the offline, non-root release proof and pip check before interruption. Scoped Sync installed the reviewed Orders up arguments and post-hook. Both old callers were then stopped for the final SQLite-consistent backup; integrity, foreign-key, and independent checksum checks passed. The retained backup/evidence directory is root-private (0700), with 0600 database/configuration copies; its location stays in the private operator receipt rather than this public journal. No restore or historical data repair ran.

Acceptance evidence	Verified result
Scoped `RunSync` Update `6ae16657a10ff54a58021307`	`Complete`, `success=true`; only Stack `orders` selected
Native `deploy-orders` Update `6ae1669ba10ff54a58021319`	`Complete`, `success=true`; exact returned ID reread, not a latest-Update guess
Web and bot image ID	Both `sha256:8599188dc1b94f00d040ada126a0908cad5ea8eb8a12a220f5041cb8fdf6511c`
Image tag and OCI revision	`homelab/orders:d7d111f139f50b0e19685abb9828f4c01ba31bbd`; matching full revision label
Entrypoint	Both `python -B /run/orders/verify-release.py`
Runtime state	Both running/healthy, zero restarts, UID:GID `10001:10001`, no host port bindings
Runtime proof	Native post-hook plus independent host check passed for both exact containers
Negative proof	Actual pre-carrier-fix image `5f218ce…` refused by the current verifier, offline and without production data mounts
Read-only application checks	`/healthz`, 3,567-carrier catalog, SQLite integrity and foreign-key checks passed
Authentication	`authentik@file` retained; unauthenticated dashboard request returned HTTP 302
Cleanup	Owned maintenance marker and shared execution intent absent; operator unit exited successfully

No manual registration, carrier correction, Gmail sync, or tracking refresh was used as a smoke test. The bot's normal scheduled polling and alerts resumed. This is local backup/deployment verification, not a new offsite restic snapshot, full restore drill, or proof of every upstream provider's health. The shared helper update can invalidate other profiles' old fingerprints; their normal refresh is required before a later authorized deployment. Other stacks were not deployed by this activation.

Limits and rollout obligations¶

The guarantee is the verified revision for the run, not an ongoing automatic latest guarantee. Source movement during a build/check window refuses validation; retry only after inputs settle. A successful Compose operation or desired-input fingerprint is not a successful post-hook receipt. There is no old-image fallback or automatic backup restore.

Maintainer, before merge: require successful current-head CI and review the revision, startup, and post-hook contracts. Local acceptance above covers dirty/ignored/hidden-index source, stale refs, remote/auth failure, metadata movement, malformed/missing manifests, changed/missing/extra runtime bytes, old-image masquerading, divergent web/bot images or health, generated-artifact fingerprinting, and unchanged behavior of other profiles.
Reviewer/operator, before rollout: merge reviewed guard source to main, quiesce competing writers, align the host homelab checkout, apply scoped ResourceSync for the Orders up arguments and post-hook, and render/bootstrap the profile. Explicit deploy-orders pulls application source, not homelab; ensure the checked-out and synced guard configuration is the reviewed one.
Data owner: follow the existing Orders migration policy: verify the required backup and coordinate web/bot against their shared SQLite state. Do not run incompatible old/new writers together; deployment approval does not authorize database replacement or restore. The existing Order Ledger recovery policy remains separate from release validation.
Operator, after separately approved rollout: record terminal Update success, both-container health/image/revision identity, runtime-byte proof, and the unchanged-input decision. The accepted activation above covers Orders only; future changes still need their own verified result.

Approved rollback needs an explicit coordinated revision, manifest, and image plan for both processes, with database compatibility/recovery reviewed separately. The normal code rollback is a reviewed revert commit on main and a new proof/build; the freshness gate intentionally rejects arbitrary older revisions. Never bypass the guard with a missing revision variable, a local/mutable fallback, or an automatic backup restore.