Skip to content

Komodo push-to-deploy (GitHub Actions relay)

How pushes to main trigger Komodo deploy-infra without public ingress to komodo.infra.realemail.app.

Why not a GitHub repo webhook?

*.infra.realemail.app resolves only on the LAN (AdGuard rewrites → 192.168.6.17). GitHub's webhook delivery IPs cannot reach Komodo directly — deliveries failed with 502 "failed to connect to host".

Komodo still polls the repo every 5 minutes; the relay gives immediate deploy on merge.

Architecture

sequenceDiagram
  participant GH as GitHub (push to main)
  participant WF as Actions (self-hosted runner)
  participant K as Komodo listener
  participant P as deploy-infra Procedure

  GH->>WF: komodo-deploy.yml
  WF->>WF: Sign payload (KOMODO_WEBHOOK_SECRET)
  WF->>K: POST /listener/github/procedure/deploy-infra/main
  K->>P: RunProcedure (PullRepo + BatchDeployStackIfChanged)
Piece Location
Workflow .github/workflows/komodo-deploy.yml
Trigger script scripts/trigger-komodo-deploy.py
Procedure services/komodo/resources.toml — deploy-infra
Runner [self-hosted, Linux, X64, homelab-lan] on infra-services (/home/someone/actions-runner, systemd)
Secret Read from host services/komodo/compose.env directly (runner IS infra-services); GitHub repo secret KOMODO_WEBHOOK_SECRET is only a fallback

Docs-only pushes are excluded via paths-ignore so mkdocs edits do not redeploy stacks.

Managed LAN runner (infra-services)

The relay requires the managed LAN-class runner on infra-services so it can resolve komodo.infra.realemail.app.

Item Value
Host infra-services (192.168.6.17)
Service github-runner@homelab-lan-01.service
Labels homelab, lan, deploy, homelab-lan-01

Check: sudo systemctl status github-runner@homelab-lan-01. Re-register after DR: GitHub Actions runner fleet / GitHub runner docs. Also used by proxmox-scan.yml and network-scan.yml.

One-time setup (done 2026-06-26)

  1. Remove any broken Settings → Webhooks entry pointing at https://komodo.infra.realemail.app/listener/... (GitHub cloud cannot reach it).
  2. Merge workflow + script to main; confirm green Komodo deploy run.

Since 2026-07-03, scripts/trigger-komodo-deploy.py reads KOMODO_WEBHOOK_SECRET from /opt/homelab/services/komodo/compose.env directly and only falls back to the $KOMODO_WEBHOOK_SECRET env var (GitHub secret) when that file is missing — see Komodo reliability audit: eliminate the GitHub Secret single point of failure. The GitHub secret still exists for portability (manual trigger from a non-infra-services machine) but the deploy pipeline no longer depends on it being correctly synced.

Manual trigger

# From a machine on LAN with infra DNS (or self-hosted runner)
export KOMODO_WEBHOOK_SECRET='…'   # from host compose.env
python scripts/trigger-komodo-deploy.py

Or: Actions → Komodo deploy (push to main) → Run workflow.

Verify

  1. Actions → Komodo deploy — step prints Komodo webhook OK: HTTP 200.
  2. Komodo UI → Procedures → deploy-infra — recent execution after push.
  3. Changed stacks redeploy; unchanged stacks skip (BatchDeployStackIfChanged).

Troubleshooting

Symptom Check
ERROR: KOMODO_WEBHOOK_SECRET not set but secret exists in Settings Empty GitHub secret value — see Empty repo secret
HTTP 401 from listener Three-way drift among GitHub, compose.env, and komodo-core — see Webhook signature mismatch
komodo-core crash loop / Mongo Authentication failed SOPS render drift vs live Mongo — see Mongo password drift
Connection refused / DNS Runner not on homelab network, or AdGuard rewrite missing for komodo.infra.realemail.app
Workflow skipped Push only touched paths-ignore paths (docs, README, grafana dashboards)
Procedure runs but pull fails Repo path must be /opt/homelab; do not use SSH in on_pull (periphery lacks ssh). Re-import resources.toml via Syncs. Mount check: services/komodo/README.md
Pull fails with "would be overwritten by merge" Host /opt/homelab has untracked files or dirty tracked files that collide with incoming main — see Host-checkout conflicts
New stack in resources.toml never deploys, older stacks work fine RunSync stage may be missing/failed — a brand-new [[stack]] needs the Sync applied once before BatchDeployStackIfChanged will ever touch it

Empty repo secret

GitHub lists secrets by name only — the value can be empty. A healthy job log shows KOMODO_WEBHOOK_SECRET: ***; a broken one shows nothing after the colon.

Re-sync from host (value only, not the whole KEY=value line):

ssh infra-services-cursor \
  'grep ^KOMODO_WEBHOOK_SECRET= /opt/homelab/services/komodo/compose.env | cut -d= -f2-' \
  | gh secret set KOMODO_WEBHOOK_SECRET --repo spadoople/homelab

Common after SOPS migration if gh secret set ran with empty stdin or captured stderr instead of the hex stdout from rotate-komodo-secrets.sh.

Webhook signature mismatch

GitHub secret, host compose.env, and komodo-core env must all match. Compare prefixes only (do not log full values):

grep ^KOMODO_WEBHOOK_SECRET= /opt/homelab/services/komodo/compose.env | head -c 30
docker exec komodo-core printenv KOMODO_WEBHOOK_SECRET | head -c 16

If compose.env differs from the running container, reconcile Mongo first (next section), then recreate core/periphery — not before.

Mongo password drift

SOPS-rendered compose.env can drift from Mongo's live password (MONGO_INITDB_* only applies on first volume init). Symptom: komodo-core restart loop with SCRAM failure: Authentication failed.

Run fix-komodo-mongo-auth.sh with the password Mongo still accepts. The init password is often visible on the mongo container:

docker inspect komodo-mongo \
  --format '{{range .Config.Env}}{{println .}}{{end}}' \
  | grep MONGO_INITDB_ROOT_PASSWORD

Do not force-recreate core alone when only the webhook secret is stale.

Host-checkout conflicts (untracked or dirty tracked files)

git merge --ff-only refuses to fast-forward when incoming main would overwrite either:

  • an untracked path already on the host (including gitignored files), or
  • a tracked file with uncommitted local edits (live patches, leftover WIP, mode bits).

Either case blocks every subsequent deploy until the checkout is cleaned (no alert — see 2026-07-03 postmortem). The 2026-08-13 Komodo deploy failure after PR #290 was the tracked-file case: host copies of ansible-pull / operator-shell already matched origin/main, but Git still aborted.

ssh infra-services-cursor 'cd /opt/homelab && git status --porcelain'

Normal GitHub-triggered deploys run scripts/komodo-safe-pull.sh /opt/homelab "$GITHUB_WORKSPACE" HEAD before sending the Komodo webhook. The lan Docker runner mounts /opt/homelab but not host SSH config, so the workflow fast-forwards from the Actions checkout instead of git fetch origin over SSH.

If a path would block the fast-forward, the script copies the working-tree file to a timestamped directory under /opt/homelab/.git/komodo-untracked-backups/, records a diff when it differs from the tip, then either removes the untracked file or git restores the tracked path back to HEAD so the merge can apply the incoming tree. Local edits on files not touched by the incoming commit are left alone.

For manual recovery on the host (where ~/.ssh/config Host aliases exist), prefer:

ssh infra-services-cursor 'bash /opt/homelab/scripts/komodo-safe-pull.sh /opt/homelab origin main'

If you must inspect by hand, diff against origin/main before touching anything:

ssh infra-services-cursor 'cd /opt/homelab && git diff --no-index <path> <(git show origin/main:<path>)'

Never rm -rf a directory under /opt/homelab without listing its contents first. Gitignored, host-local files (for example a service .env) may live in the same directory. This exact mistake (deleting services/litellm/.env during cleanup) is documented in the postmortem above.

After recovery, re-trigger the deploy (Actions → Komodo deploy → Run workflow).

If this recurs, find the writer: live edits on tracked files in /opt/homelab (apply those on a branch instead), a manual docker compose test that dropped untracked files, or a deploy interrupted mid-checkout. The script archives blockers so deploy can proceed; it does not stop the next live edit.

After Komodo SOPS rotation or render

  1. Re-sync GitHub (step 1 under One-time setup).
  2. Full rotate via rotate-komodo-secrets.sh: recreates core/periphery, updates Mongo, prints hex for gh secret set.
  3. Render-only (no rotate): reconcile Mongo with fix-komodo-mongo-auth.sh, then recreate core/periphery.
  4. Confirm: Actions → Komodo deploy → Run workflow → Komodo webhook OK: HTTP 200.