Skip to content

PR 353 — OneUptime AI evidence (owner runbook)

PR: https://github.com/spadoople/homelab/pull/353

PR 352 is merged, production is on 12.0.27, and the live observation remained healthy with zero app, ingress, Probe, or Runner restarts. PR 353 merged and its runtime pieces were accepted on 2026-09-05.

Program map: OneUptime v12 merge program.

This PR adds a read-only evidence bridge, a disabled-by-default adapter seam, LiteLLM digest/alias updates (including local gpt-oss:20b), and the qualification harness. It does not turn on incident AI in OneUptime and does not authorize broker mutation.

What merge actually deploys

Piece On merge to main Default
LiteLLM config.yaml / image pin Yes, via Komodo deploy-infra (litellm is auto-deploy) Local alias oneuptime-incident-local-gpt-oss-20b exists; hosted OpenAI is not required
Incident adapter code Yes, via oneuptime-incident-adapter auto-deploy EVIDENCE_BRIDGE_URL stays empty → evidence client is off
Evidence bridge Compose In git only Not a Komodo stack. No Traefik, no host ports. Deploy later, reviewed
Qualification harness In git (services/oneuptime-ai-eval/) Not a runtime

Merging 353 is safe for production incident ingest as long as you leave EVIDENCE_BRIDGE_URL empty.

Progress

# Task You Agent Status
0 PR 352 observation green Confirm v12 UI and seven-service health Confirm smoke-test still passes Done 2026-09-05
1 Merge PR 353 Review and merge Let deploy-infra update LiteLLM + adapter Done 2026-09-05
2 Confirm evidence still off Optional spot-check: new incident has no evidence JSON block Confirm rendered URL empty and adapter healthy Agent checks done
3 Optional: deploy bridge (later) Generate matching tokens; approve internal-only compose Render SOPS, docker compose up for the bridge, set adapter URL Off until you say so
4 Do not enable OneUptime AI agents / hosted OpenAI Leave Runner AI commands off; ChatGPT Plus is not an API key — Locked

1. Merge

Preconditions:

  • Production app image is oneuptime/app:12.0.34@sha256:….
  • #353 is retargeted onto main (or onto merged 352) and CI is green.
  • You are not trying to turn evidence on the same afternoon as v12 cutover unless 352 observation is already boring.

Merge. Ordinary deploy-infra may restart LiteLLM and the incident adapter. That is expected. It must not be used as a substitute for deploy-oneuptime.

2. Verify fail-closed evidence

After adapter deploy:

  • services/oneuptime-incident-adapter/config.env still has EVIDENCE_BRIDGE_URL= empty.
  • A normal Alertmanager → adapter → OneUptime incident still creates/updates without waiting on Prometheus/Loki.
  • Verify direct destinations only for matching alerts: Discord requires notify=page; ntfy requires critical service=os-patching. continue: true evaluates later sibling routes, not universal failover. Allowlisted runner notify=ticket alerts have no direct Discord fallback; do not change their labels to satisfy this check. Use the receiver matrix and separate delivery acceptance. Adapter rollback needs a reviewed ticket fallback or an explicit owner-approved notification-loss window; clearing the evidence URL is not a routing rollback.

If an incident description contains ### Read-only evidence packet, the bridge was enabled accidentally. Clear the URL, re-render adapter compose.env, redeploy the adapter.

3. Later: live read-only evidence (optional, separate window)

Only after you explicitly want runner-domain packets on first projection.

  1. You: generate one shared token (openssl rand -base64 48); put the same value in:
  2. services/oneuptime-evidence-bridge/.env.sops.yaml → EVIDENCE_BRIDGE_TOKEN
  3. services/oneuptime-incident-adapter/.env.sops.yaml → EVIDENCE_BRIDGE_TOKEN
  4. Agent: encrypt with repo SOPS recipients; render both compose.env files; start the bridge on the internal Docker networks only (no Traefik).
  5. Set adapter EVIDENCE_BRIDGE_URL to the internal bridge URL (not a public hostname). URL without token must fail adapter startup — that is the guard.
  6. Deadline stays ≤ 30 seconds. Evidence fails open; ingest must not block.
  7. Accept only github-runners plus the four enumerated runner alert names. Unknown domains must 4xx.

Do not put the bridge on Traefik. Authentik is not used (machine-only).

4. Models (locks)

  • Selected path: workstation Ollama gpt-oss:20b via LiteLLM alias oneuptime-incident-local-gpt-oss-20b (16K context, low reasoning, 512 output tokens). Qualification already passed (36/36 valid, 85.7% eligible accuracy, 100% unsafe abstention, zero fabricated targets).
  • Hosted oneuptime-incident-frontier / balanced / economy aliases are inputs to the harness, not a production ranking. They need a real OpenAI API account with credits. ChatGPT Plus/Pro does not count.
  • Do not enable OneUptime Runner ONEUPTIME_RUNNER_ENABLE_AI_COMMANDS.
  • Do not merge PR 354's mutation flag as part of this PR.

Stop / abort

Leave evidence off by emptying EVIDENCE_BRIDGE_URL and redeploying the adapter. Stop the bridge compose without -v (it is stateless; no backup required).