Skip to content

Komodo edge recovery and relay hardening — 2026-10-29

Accepted edge rollout

The owner authorized a fresh edge-only deployment of merged main fabb01b after PR #531. This superseded the global dispatch hold, not the exact-run admission guard for orphan 40469274207. The guard/verifier remained intact; the original run still reported queued with zero jobs. GitHub Support ticket #4922716 was closed on self-service support eligibility, not because the orphan was reconciled.

The Docker CI runner was recovered first through the existing exact-runner fleet workflow: resume 40826054412 and qualification 40826228770 succeeded. The previously queued Phoenix/OneUptime jobs and merged-main Lint completed successfully. No bulk runner restart or routing-label workaround was used.

Deployment preflight established:

  • All 56 managed resources were readable and idle; ResourceSync had no resource, variable, user-group or deployment changes.
  • Deployed-to-desired edge Compose differences were image references only.
  • Canonical rendering matched all four shared on-pull env files byte-for-byte.
  • The pending relay journal was absent and its persistent lock available.
  • An owned maintenance marker and paused timers excluded future Ansible starts. The existing apply was still waiting on the check-mode lock; it was stopped before the check-mode process. No active configuration apply was deliberately interrupted.
  • A private rollback snapshot and runtime baseline were retained on infra-services under /var/lib/homelab/edge-rollout-20261029T061136Z.

First dispatch 40827901120 stopped before stack deployment: the Periphery preparation hook required a missing local ghcr.io/moghtech/komodo-periphery:2.3.4 helper. The exact failed Procedure and PullRepo Updates were terminal. The prescribed helper was cached at digest sha256:fb4b52961fcce7a31e3776930e422967b6640f653de876f59f26dea243e29beb, without restarting Core/Periphery. Preparation then passed through the actual Periphery path; rendered credentials remained byte-identical and no workload container changed.

Deliberate retry 40828933347 completed the native edge Procedure:

Exact Update Operation Verified outcome
6ae2e80ea10ff54a58025af0 RunSync Complete, success=true
6ae2e82ba10ff54a58025af7 RunProcedure: deploy-reviewed-images-edge Complete, success=true

The Actions conclusion remains failure. Traefik's restart interrupted the old relay's HTTPS Update polling. Komodo continued the already-submitted execution. Once API access recovered, the exact pending ID, operation and target were checked under the host lock. The existing journal recovery code cleared intent only after successful terminal completion was verified. No replacement execution was submitted to manufacture a green result.

flowchart TD
    A[Owner-approved fresh edge dispatch] --> B[External Sync verified]
    B --> C[Pull and prepare reviewed main]
    C --> D[Conditional edge stack batch]
    D --> E[Traefik restart interrupts relay polling]
    E --> F[Keep exact pending intent]
    F --> G[Verify same native Update under host lock]
    G --> H[Runtime checks and restore timers]

Runtime evidence:

  • Traefik reports 3.7.16. All four edge containers match the reviewed digests.
  • Homepage, AdGuard and Unbound are healthy; Traefik is running without a configured container healthcheck.
  • Public DNS succeeds over UDP/TCP, the Homepage LAN rewrite is correct, and the existing AdGuard smoke test passes.
  • Homepage's backend returns HTTP 200. Homepage, Traefik and AdGuard HTTPS routes return the expected 302 to Authentik with TLS verification enabled.
  • Only the four edge workload containers changed; normal runner recycling is recorded separately. No non-edge workload or Core/Periphery container changed.
  • Both Ansible timers were restored active; the owned maintenance marker was removed; pending intent is absent. The manual-only workflow remains enabled and the orphan guard remains in force. Private snapshots/receipts were retained, not deleted.

See the PR deployment receipt. Documents/finance, authenticated user-session acceptance, an unchanged second rollout, and the optional Core dispatch token were not accepted by this operation.

Source hardening follow-up

At the source-hardening checkpoint these changes were implemented but not yet merged or live-qualified. PR #532 subsequently merged as cfaf155; the post-merge qualification below records normal-path acceptance separately from the earlier rollout.

Exact Update polling

Only GetUpdate for a recorded execution ID retries transient transport failures, request-local timeouts, incomplete HTTP bodies, or HTTP 502/503/504. Six consecutive retries use bounded 1/2/4/8/8/8-second backoffs within the existing overall deadline. Validated reads reset the consecutive-error budget. The host lock and original pending record remain unchanged while waiting.

Execute requests are never retried, even for those errors. Unknown execution IDs, authentication/authorization errors, missing Updates, redirects, TLS failures, malformed complete JSON, oversized responses and mismatched IDs/targets remain fail-closed. Resource-discovery and busy-state reads do not acquire retries. Late worker responses cannot replace a newer request's result or alter the journal.

Checkout preparation

The live helper checked roughly 18,600 ignored paths, mostly .venv, with one Git process per path, taking several minutes. The replacement resolves an immutable incoming commit and uses one target-tree inventory plus checked NUL-delimited untracked/ignored and staged/unstaged inventories. Git process count no longer grows with the number of ignored files.

All archive copies finish before any removal/restoration. Working-copy originals and symlinks are preserved without optional .diff sidecars. Incoming-conflicting paths with distinct staged and working versions, structural directory conflicts, and intersecting collapsed directory entries fail closed for manual resolution. The final --no-overwrite-ignore prevents silent ignored-file overwrites missed by the inventory or introduced afterward. This is not a transactional checkout rollback; independent writers must still be excluded and originals retained for recovery.

Validation and next gates

Focused tests cover proxy interruption during Sync/Procedure polling, retry budgets, overall and request-local deadlines, late responses, permanently invalid responses, secret-safe errors, execute non-retry, fixed Git invocation counts with 4,000 ignored files, literal filenames, symlink targets, nested ignored repositories, producer and archive failures, staged-byte preservation and concurrent ignored blockers.

Local validation completed:

  • uv run python -m unittest tests.test_trigger_komodo_deploy: 101 tests passed.
  • python3 -B tests/test_komodo_safe_pull.py -v under Linux/Bash 5: 19 tests passed.
  • Full Python discovery in the offline, unprivileged Linux validation container: 2,395 tests, 52 skipped; passed. The source mount was read-only, the disposable checkout used noatime, and no Docker socket or production credential was mounted.
  • Documentation contract suites: 115 tests passed; Markdown fix/check and uv run mkdocs build --strict passed.
  • Applicable staged pre-commit hooks, diff checks and Gitleaks passed.

PR CI is an additional gate, not a substitute for those local checks. After merge, separately qualify the new relay behavior before expanding rollout scope. The helper-image cache prerequisite remains explicit; this source change does not add an implicit image pull or extend privileges.

Post-merge unchanged edge qualification

Run 40836050724 on cfaf155 finished SUCCESS in 2m49s. The existing Ansible apply was allowed to finish successfully before dispatch; it was not interrupted. Exact Sync 6ae2f6eda10ff54a58025d1b and Procedure 6ae2f706a10ff54a58025d22 both completed successfully. The edge conditional batch took about 0.63 seconds.

All 48 workload container IDs and start times remained unchanged; normal runner recycling is excluded. Shared rendered env bytes, DNS/HTTPS checks and the Homepage backend remained valid. Both timers were restored, the owned maintenance marker removed, pending intent absent, and the orphan guard verified. See the qualification receipt. This qualifies normal execution/no-op selection, not a live fault-injection test of proxy-outage recovery or a documents/finance rollout.

Remaining-cohort preflight: Stirling downgrade prevented

Before the requested remaining rollout, read-only runtime inspection found healthy Stirling-PDF 3.2.2-fat, running from latest-fat, with digest sha256:b63cd013fcd242210b137d6ac5aeb9fcf4c5cb9d4e004b8a3e3185fcd57d3c9e. docker buildx imagetools inspect docker.stirlingpdf.com/stirlingtools/stirling-pdf:3.2.2-fat independently returned that OCI index digest. This was a metadata request, not an image pull or container change.

Git's desired pin was 2.14.3-fat, selected from its stale 2.14.0-fat source baseline; Komodo's deployed Compose also recorded 2.14.0, not the actual floating-tag runtime. The documents Procedure would therefore perform an unintended major downgrade. No deployment was dispatched. Read-only database/version and backup-metadata checks are not fresh-backup or restore acceptance.

The source correction pins the already-running 3.2.2-fat bytes and updates the service README/proposal history. A regression floor prevents reintroducing the known 2.x downgrade while allowing future patch and digest updates. Automatic proposal scope and same-minor selection are unchanged. Review/merge of this corrective pin is required before resuming documents-first rollout; no runtime override, Procedure bypass, backup deletion, timer change or documents/finance deployment accompanies this correction.

Correction validation: 154 image-update tests and the full offline Linux suite (2,396 tests, 52 skipped) passed. Documentation/image-procedure checks (144 tests), Markdown fix/check and strict MkDocs build also passed. These source checks and the registry metadata comparison do not replace the remaining backup and runtime gates.