2026-09-07 Remediation broker live, mutation off

TL;DR

The OneUptime remediation broker is running on infra-services with BROKER_ENABLE_MUTATION=false. Deny-all verifiers rejected a well-formed start_ephemeral_runner request and wrote an audit row. Exact-label wake proof for gha-canary-a already passed. Mutation stays off until you explicitly enable it.

Phase status

Gate	Status
Monitoring canary	Healthy (remediation_canary_info=1)
gha-canary-a wake proof	Passed (run 34017669466)
Idle-offline (no page)	Passed; PR #373
Broker deploy, mutation false	Live; PR #374
Deny audit	rejected / precondition_failed for homelab-ephemeral-canary-01
Kill switch	Clear (enabled=false)
Verifier HTTPS origins	Unset on purpose (deny-all)
Mutation canary	Off until you set BROKER_ENABLE_MUTATION=true

flowchart LR
  W["Exact-label wake<br/>gha-canary-a"] --> I["Idle-offline<br/>no page"]
  I --> S["SOPS + render<br/>compose.env 0600"]
  S --> B["Broker healthy<br/>mutation false"]
  B --> D["409 deny-all<br/>audit row"]
  D --> V["Owner: verifier URLs<br/>or keep deny-all"]
  V --> M["Owner: mutation true<br/>plus 24h canary"]

What shipped

• Broker and dedicated PostgreSQL are healthy on the private Compose networks. No host ports, no Traefik.
• First authenticated request was rejected by unconfigured verifiers and stored in remediation_attempts.
• Tier-1 logical backup and isolated restore drill for that audit database.

Known gaps / drift

• Komodo webhooks authenticate then run asynchronously; they are not a reliable Compose substitute for the broker (not a Komodo stack).
• BROKER_VERIFIER_TOKEN and the four read-only HTTPS origins remain unset.
• Idle-runner restart stays validation-only.

What remains

1. Owner: supply verifier HTTPS origins (and token if those APIs need it), or keep deny-all.
2. Owner: only after that, set BROKER_ENABLE_MUTATION=true on the host env for a 24-hour canary. Do not flip it from a request body.
3. Komodo procedure webhooks: diagnosed 2026-09-16; see 2026-09-16 check-in.
4. Owner: OneUptime provisioner API key and optional SMTP From Email remain from the v12 cutover list.