LLM Observability - LiteLLM + Phoenix

This stack has two jobs:

Tool	Host	URL	Daily job
LiteLLM	infra-services	https://litellm.infra.realemail.app	One OpenAI-compatible gateway, virtual keys, budgets, usage, Prometheus metrics
Phoenix	prox LXC 124	https://phoenix.infra.realemail.app	Trace inspection, prompt/RAG debugging, evals, datasets, experiments

Use LiteLLM for normal traffic. Open Phoenix when a request behaved strangely, you are tuning a prompt chain, or you need evidence about what happened inside a multi-step LLM workflow.

Current status — 2026-10-26: the explicitly authorized production rollout of main 924079477549e3962a3f4b2c6c9073aeb7fdaf72 succeeded. LiteLLM 1.106.4 is serving, and bounded live local chat/embedding routing passed. See the production result for evidence, earlier failures and the still-required scheduled-backup/dirty-checkout follow-ups. This is not proof of canonical Komodo path convergence or a production rollback restore.

Mental Model

Most clients should only know about LiteLLM:

client -> https://litellm.infra.realemail.app/v1 -> provider/backend
                                              |
                                              +-> Prometheus metrics
                                              +-> Phoenix traces

Phoenix is not another gateway. It is the trace workbench behind the gateway. A successful export is not proof that a model call reached the correct project. The earlier 2026-10-26 audit found per-key routing broken on LiteLLM 1.98.0. The remediation below is now deployed, with exact request attribution verified for the tested local chat and embedding paths.

The intended split is model-call spans in each application's project, with HTTP/auth/database operational spans intentionally separate in fallback project litellm. Per-key routed model spans become new trace roots linked to, not parented under, the gateway request. Matching the application's project name alone does not produce an automatic application E2E trace. Retrieval, tools, agent steps, and other workflow spans require application instrumentation.

Day In The Life

One-time setup for a new tool

1. Open the LiteLLM Admin UI at https://litellm.infra.realemail.app/ui.
2. Create a virtual key for the client, such as comfyui, cursor, or homelab-script.
3. Give the key a budget/rate limit that matches the blast radius of that tool.
4. Configure the client:

Base URL: https://litellm.infra.realemail.app/v1
API key:  <LiteLLM virtual key>
Model:    one of the names from /v1/models, for example ollama-llama3.1

The Authentik browser session only protects the UI. API clients authenticate with LiteLLM virtual keys.

Phoenix project per virtual key

Phoenix can hold many projects on one collector. The desired assignments for existing virtual keys are in services/litellm/virtual-keys/phoenix-projects.yml; aliases and project names are exact and case-sensitive:

Virtual-key alias	Intended Phoenix project
Captain_Kangapoo	Captain_Kangapoo
captain_comfyui	comfyui-ollama
telegram-companion	telegram-companion
paperless-litellm	paperless-litellm
litellm_test	litellm_test

telegram-companion (hyphen) is canonical because it matches the active app exporter. Historical telegram_companion (underscore) projects/traces are not renamed, moved, or deleted. The map does not provision keys or projects.

From the repository root:

python3 scripts/configure-litellm-phoenix-projects.py          # offline metadata preview
python3 scripts/configure-litellm-phoenix-projects.py --check # live metadata, read-only

The default preview needs no credentials or network. --check requires LITELLM_MASTER_KEY and exits nonzero on metadata drift or validation failure; it does not test trace routing. --config selects another YAML map, and --base-url selects the LiteLLM root URL without /v1 (default: https://litellm.infra.realemail.app). The public /key/* and /team/* paths are behind Authentik; a bearer token alone does not bypass browser middleware. Use the container-local admin route in rollout step 5, not a new public route. The script refuses redirects, including an Authentik login redirect.

--apply is a separate, manual opt-in only after the approved image rollout (see below). The production apply changed only telegram-companion from the underscore to the hyphen project; all five assignments then checked drift: false. The updater paginates key discovery, requires unique matching aliases, preflights all mapped keys and team conflicts before writing, preserves unrelated metadata, skips unchanged keys, and verifies metadata readback after updates. It writes both phoenix_project_name and phoenix_project_name_override. A key override beats an ordinary team project, but not a conflicting team override; that conflict is refused, not silently fixed. The audit found no conflicting team Phoenix metadata.

Updates are not transactional: concurrent edits or partial writes remain possible. Inspect live state before retrying a failed apply. Omitted aliases retain their metadata; omission does not reset a key to fallback. The unmanaged prometheus-scrape key has no project metadata. Deployment does not automatically reconcile this administrative map.

Routing requires the guarded LiteLLM implementation as well as LITELLM_OTEL_V2=true and Phoenix's OTLP/HTTP x-project-name support (15.5.0+). The flag and correct metadata alone did not enable routing on 1.98.0. Prometheus scrapes of LiteLLM /metrics are excluded from server tracing by OTEL_PYTHON_FASTAPI_EXCLUDED_URLS; services/litellm/otel_quiet.py suppresses health/metrics/UI auth phase noise. That filter is baked into the image: rebuild and recreate the proxy after changes, subject to the rollout gates below. It does not suppress all operational spans or remove historical auth /metrics traces.

Normal usage

• Use LiteLLM as the base URL for OpenAI-compatible tools.
• Check LiteLLM UI for per-key spend, model usage, and key management.
• Check Grafana -> Homelab -> LiteLLM for request rate, token throughput, error rate, and cumulative spend.
• Use Phoenix only when you need to inspect a run: inputs, outputs, timings, errors, or eval candidates. Tool/retriever spans are available only when the application instruments them; the gateway does not create them.

When something looks wrong

Start in LiteLLM if the problem is "the request failed" or "this key/model is too expensive." Start in Phoenix if the request succeeded but the answer was bad, slow, incomplete, or hard to explain.

For prompt/RAG work, instrument the application workflow as well as routing model calls through LiteLLM. Then the useful loop is:

1. Send the request through LiteLLM.
2. Open Phoenix and inspect the application trace and linked gateway model traces.
3. Find the span where quality changed: prompt construction, retrieval, tool call, model call, parser, or post-processing.
4. Adjust the prompt/code/data and run it again.
5. Promote good examples into a dataset or eval when repeated testing matters.

E2E Tests

A gateway smoke test checks API availability; the opt-in verifier checks bounded model-span attribution across projects. Neither replaces database migration rehearsal or application workflow testing. The production acceptance made exactly four client inference requests: two local chats and two local embeddings. This does not certify every current client's workflow.

Quick Gateway Smoke

From any machine that has a LiteLLM virtual key:

curl -sS https://litellm.infra.realemail.app/v1/models \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY"

Expected: HTTP 200 and a model list. Current local Ollama models: ollama-llama3.1, ollama-north-mini-code, ollama-qwen3.6-27b-mtp, and ollama-qwen3.6-27b-heretic-mtp.

Optionally send one completion, only with approval for model costs and synthetic prompt/completion retention. The example model must be allowed for your virtual key; substitute an operator-selected allowed chat model if it is not:

curl -sS https://litellm.infra.realemail.app/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama-llama3.1",
    "messages": [
      { "role": "user", "content": "Reply with exactly pong." }
    ],
    "max_tokens": 20
  }'

Expected: HTTP 200 with a chat completion. If this fails while /v1/models works, the configured backend may be offline. The ollama-* models point at the owner's Personal-VLAN workstation (192.168.3.17:11434), so they only respond while that workstation and Ollama are running.

Full LiteLLM -> Phoenix Trace Test

Use scripts/verify-litellm-phoenix-routing.py, not the former master-key request plus an arbitrary recent trace count. That check proved neither request attribution nor project isolation. Run the verifier from the repository root on an operator host with access to the LiteLLM API and Phoenix read API.

The required --config is an operator-supplied JSON file, not the mapper's YAML. This inline schema example uses known alias/project pairs; it is not a ready-to-run model allowlist or a config file created by this remediation:

{
  "litellm_base_url": "https://litellm.infra.realemail.app/v1",
  "phoenix_base_url": "http://192.168.6.124:6006",
  "phoenix_api_key_env": "PHOENIX_API_KEY",
  "fallback_project": "litellm",
  "cases": [
    {
      "key_env": "TELEGRAM_COMPANION_VIRTUAL_KEY",
      "key_alias": "telegram-companion",
      "project": "telegram-companion",
      "model": "REPLACE_WITH_OPERATOR_SELECTED_ALLOWED_CHAT_MODEL_FOR_TELEGRAM_COMPANION"
    },
    {
      "key_env": "COMFYUI_VIRTUAL_KEY",
      "key_alias": "captain_comfyui",
      "project": "comfyui-ollama",
      "model": "REPLACE_WITH_OPERATOR_SELECTED_ALLOWED_CHAT_MODEL_FOR_CAPTAIN_COMFYUI",
      "max_tokens": 16
    }
  ]
}

All fields are required except fallback_project (default litellm) and each case's max_tokens (default 16, positive integer). Unknown fields are rejected; base URLs may already end in /v1. Replace both model placeholders with operator-selected models authorized for the corresponding key and supporting /v1/chat/completions and max_tokens. The tool does not discover an allowlist; offline schema validation does not establish that a model is allowed or exists.

Supply at least two distinct genuine virtual keys: different environment references, aliases, and credential values, spanning at least two distinct expected projects. Use a separate Phoenix read credential with access to all projects; enumeration cannot detect projects hidden by authorization. Do not use the LiteLLM master key for model calls. MASTER-named references and values matching MASTER-named environment credentials are refused, but renaming an opaque master credential cannot make it safe or reliably detectable offline.

Inject credentials through the approved secret mechanism, not JSON or command line literals. Do not dump .env, run env/printenv, or enable shell tracing (set -x). The LAN HTTP URL above sends bearer credentials in plaintext; use only the approved trusted network path, or an approved HTTPS API route. The browser's Authentik session is not collector/API authentication; redirects are refused and TLS verification is not disabled.

Commands below use /path/to/cases.json as the operator's chosen file path:

python3 scripts/verify-litellm-phoenix-routing.py --help
# Default: offline schema validation, no credentials or network needed.
python3 scripts/verify-litellm-phoenix-routing.py --config /path/to/cases.json
# Only after rollout approval, metadata apply, and explicit consent:
python3 scripts/verify-litellm-phoenix-routing.py --config /path/to/cases.json \
  --allow-model-calls --poll-timeout 60

--allow-model-calls consents to one non-streaming chat completion per case. Provider charges may apply; LiteLLM, the provider, and Phoenix may retain the unique synthetic prompt and completion. No private prompt is supplied. Model requests are never retried, including ambiguous transport failures; rerunning incurs new calls and possibly new costs. This verifier makes no streaming or embeddings calls and does not change key/team metadata or deploy anything.

Verification requires each exact completion id to match gen_ai.response.id on a model-call span with the exact key alias (litellm.metadata.user_api_key_alias) and expected project. Missing or duplicate response IDs and missing attribution attributes fail closed. Time, model name, traceparent, an LLM label alone, or any recent trace is not a substitute. The provider's model name need not equal the LiteLLM model alias.

The verifier paginates Phoenix /v1/projects and each project's spans, searches every listed project plus expected projects and fallback litellm, and refreshes project enumeration each poll. Wrong-project model spans, including fallback emissions, fail verification. Correlated HTTP/server, auth, and database spans are counted separately as operational noise even if labeled LLM; unknown correlated spans are not silently excused.

Polling lasts the full --poll-timeout window (default 60 seconds), even after correct spans arrive, to catch late wrong-project duplicates. Defaults are --poll-interval 2, --request-timeout 30, and --clock-skew-seconds 30. Retryable GET failures are retried only within the window. Incomplete final sweeps, malformed pagination, or scan limits fail verification: incomplete coverage is inconclusive, not proof of misrouting or success. Investigate API latency and adjust timeout/interval headroom before an approved rerun; a longer window cannot guarantee the final sweep fits or exclude arbitrarily late exports.

Output contains safe case metadata, SHA-256 response-ID fingerprints, actual trace/span IDs, projects, and noise counts—not raw response IDs, prompts, completions, request markers, credentials, or HTTP bodies. The tool writes no files. Exit 0 with status: not_run means offline validation only; exit 0 with status: passed means successful bounded observation. Exit 1 is verification failure, 2 is invalid configuration/CLI, and 130 is interruption. The production run used exact chat response-ID and embedding x-litellm-call-id attribution. A corrected, read-only audit accepted normal OTel OK/UNSET success statuses while still rejecting ERROR; it reused the two existing chat responses rather than repeating inference. See the production failure history; do not assume the chat-only verifier above covers embeddings.

Historical export-only evidence: on 2026-07-05, before the callback registration fix in services/litellm/config.yaml, /v1/models, chat completion, and Phoenix /v1/projects returned 200, but Phoenix returned zero traces. After the callback fix, the 2026-07-28 smoke test returned HTTP 200 and pong; project litellm returned trace 8d077eb51bdf502432daf02bf47d555f with three spans, two LLM spans, all statuses OK, and 64 total tokens. This established export success, not per-key attribution, project isolation, or completion of this remediation.

Routing remediation rollout

Production rollout and bounded live routing acceptance passed on 2026-10-26. This followed explicit owner authorization, the password-preserving fix in merged PR #509, and the earlier isolated rehearsals. The initial failure at d839f37 and patched success at 9623e82 remain separate historical evidence; both original JSON reports are unchanged. The upstream 1.106.4 upgrade alone was insufficient, and this was not a first-attempt production pass.

Production result: 2026-10-26

The compact sanitized report binds the exact source/image/backup hashes to the outcome. This documentation uses only the five approved safe parent-local result logs and the owner's rollout summary/final read-only host evidence; this documentation update did not access raw logs or production hosts.

Active release and preserved database. Main 924079477549e3962a3f4b2c6c9073aeb7fdaf72 was deployed from the root-private /var/lib/homelab/rollouts/litellm/924079477549e3962a3f4b2c6c9073aeb7fdaf72/source/services/litellm (hereafter R is the release root above source). The running Compose working directory remains R/source/services/litellm, while canonical Komodo uses /opt/homelab/services/litellm.

The shared checkout's historical HEAD was 034fe1d at 10:18 on 2026-10-26; final read-only evidence shows /opt/homelab HEAD at 924079477549e3962a3f4b2c6c9073aeb7fdaf72. Another actor advanced HEAD: this rollout ran no fetch, checkout, reset or stash and made no shared-worktree writes. Final git status --short output matches the recorded before-output byte-for-byte, with the same dirty paths. Worktree contents were not hashed, so this is not proof of byte-identical file contents. Current HEAD alone does not establish canonical deployment-path convergence.

The AMD64 wrapper is homelab/litellm:rollout-9240794, image ID sha256:c2a71ca38564a684b38277724b2c39794a1b1a885478a9996eaa0b1b66f781a5, LiteLLM 1.106.4, with patched handler SHA-256 45b4147265096328520271c5ee0bdb7ae715c221c4ff2374d4c31fd8e696a015. Both build guards passed on an unchanged retry after the initial 60-second watchdog timeout under load. No deadlines or gates were bypassed.

The original PostgreSQL 16.15 Alpine container and f4610795… image were retained, with no recreation or password rotation. The unused Anthropic placeholder was removed from the encrypted source and new proxy environment; the old database container's environment was intentionally unchanged. The active release's canonical-input/HMAC checks passed. The proxy started at 11:41:30 UTC, was ready at 11:43:02 UTC, and remained running with restart count 0 at the final read-only inspection approximately 15 minutes later. Public https://litellm.infra.realemail.app/v1/models returned 200 / 19 aliases.

First cutover blocked safely; second passed. The official backup producer set PGSERVICE=''; libpq treated that as an empty service name and failed before staging or migration. The old proxy automatically restarted, was health-checked, and its 75-table schema was verified unchanged. Narrow fix a8041fd21ea7051a9cbea5daab9be01c98df19b9 (env -u PGSERVICE) was reviewed and passed real PostgreSQL 16 tests. Only the temporary R/backup-source used it for the one-off canonical producer. This was not a backup guard bypass: read-only, storage/headroom, identity, checksum and all other gates remained. PR #511 is still the permanent merge/install follow-up: main 9240794 still contains the PGSERVICE bug, and the scheduled installation still lacks the fix. Advancing the shared checkout's HEAD to that main commit does not install #511.

The second cutover retained a fresh quiesced immutable generation at /var/backups/postgres-logical/litellm/20261026T113657.155710Z-d124e7eb6ea5bb486b99873f59eb5cc4/: litellm.dump, 9,029,830 bytes, SHA-256 286fb6b3376d0a28c7c0f8afa6e81747c8f846dfa287102b340a944681710e14, with manifest and SHA verification. It is local-only; no offsite proof. Production migrations exited 0, advancing 157 → 213 (56 added), with all 213 checksums valid. All 74 original application-table counts and original-column digests were preserved at migration verification, using only the reviewed LiteLLM_ShadowEvalJob.api_key_id → target_id comparison alias. There were zero invalid indexes, zero unvalidated constraints, and two request-log indexes.

The old wrapper sha256:e92b677f4bc0bd3903fb94c37f26bf593ab2b8d54d3c1c52b5e3896f5932c5b0 is retained as homelab/litellm:rollback-pr509-20261026, plus a 391,444,992-byte R/rollback/image.tar and private old environment/configuration. No secret values are published. No production rollback/restore was performed; earlier restore acceptance was isolated. Never treat the shared checkout as a verified rollback or downgrade the old image onto the migrated database.

Metadata and bounded live routing. All five assignments in the table above read back drift: false. Only telegram-companion changed underscore → hyphen; Captain_Kangapoo, captain_comfyui → comfyui-ollama, paperless-litellm and litellm_test were unchanged. The original six active keys remain. Exactly four client inference requests, all HTTP 200, were made:

• Two ollama-llama3.1 chats on the local GPU workstation (192.168.3.17).
• Two companion-archive-embedding calls on the Mac (192.168.3.37), each returning 768 dimensions.

Both models were already installed; cold loads were normal. No model pulls or paid-provider requests occurred. Each operation used the existing telegram-companion and litellm_test projects, with no new projects. Chat response IDs and embedding x-litellm-call-id headers matched exactly (with only SHA-256 fingerprints published) across all 10 visible projects for full 60-second windows, with zero wrong-project model spans. This cannot exclude authorization-hidden projects or arbitrarily late exports.

Acceptance fixture failures are retained, not erased. An initial attempt made zero inference requests because it incorrectly expected /key/info to echo the token hash; independent alias/hash binding through /key/list fixed readback. The next attempt made the two successful chats, but an overstrict status audit rejected normal OTel UNSET spans. The corrected audit accepts OK or UNSET, still rejects ERROR, and audited those prior responses read-only (zero inference requests, POSTs or metadata writes), not by repeating chat calls. That audit and the final two embedding calls passed. Across fixture corrections five temporary keys were created; all were deleted and verified, leaving zero active temporary keys. Temporary public helpers were removed.

Remaining boundaries and follow-ups. Merge/install #511 for scheduled backups; current main 9240794 does not contain that fix. Review dirty /opt/homelab without hard-resetting user changes, then refresh/check inputs at /opt/homelab/services/litellm and verify canonical/periphery convergence before future Komodo deployment. Current Git HEAD is not proof of convergence with the running Compose directory in R; no host resource sync or broad deployment was done. The active pinned release is already serving; these are follow-ups, not a claim that rollout is still pending. Historical Phoenix projects remain intact. The collector was not recreated; the earlier UI observation of 20.26.0 is still not runtime digest verification. Gateway HTTP/auth/database spans intentionally stay in litellm; project routing does not automatically stitch application trees. Live streaming, fallback and all current application workflows are not certified by these four requests.

Earlier renderer stop - historical

After PR #507, the earlier production attempt stopped before cutover: the canonical renderer's default restricted value/--url-safe validation rejected the existing valid complex database password and an unused ANTHROPIC_API_KEY placeholder. That attempt caused no downtime, production migration or key-metadata apply; it was not a successful rollout. The user explicitly authorized rollout, preserving the existing password and removing only the unused placeholder. PR #509 supplied the preserving fix before the successful execution above. This documentation grants no new authorization; do not weaken placeholder checks, rotate the password or use ALTER ROLE.

Audit and prepared image evidence

The historical pre-rollout live audit on 2026-10-26 found:

• LiteLLM 1.98.0, with OTel v2 enabled, ignored per-key Phoenix projects. Trace c503991081b7cfb273415633d509590b for key telegram-companion and model companion-archive-vision landed in litellm. Teams had no conflicting Phoenix metadata; changing team assignments is not the remedy.
• Phoenix UI reported 20.26.0, with 10 projects, including the telegram_companion / telegram-companion duplicate. The underscore project's latest activity was 2026-09-05. Keep that historical data intact.
• Phoenix's running container digest was not verified because SSH access was denied. The UI version is not evidence of the runtime digest.

Prepared source pins (not proof of production deployment):

Source	Image
services/litellm/Dockerfile	ghcr.io/berriai/litellm:v1.106.4@sha256:c169fb3adc514fcf210377c203f3de406d54f10bfdacf13948a4a839db9af7de
services/phoenix/compose.yml	arizephoenix/phoenix:20.26.0@sha256:095f498723ec1fa3b55592af83d5a1e3bf026ba15a4a44e325f20dfdbb71452a

Phoenix is pinned to the observed version; no Phoenix version upgrade is intended. Verify its runtime digest when authorized access is available rather than assuming the prepared pin is already running.

services/litellm/check_phoenix_routing.py exercises the real installed callback factory, auth-metadata adapter, logger lifecycle, and exporter against a local OTLP/HTTP receiver. It checks project headers, fallback, two project routes, override precedence, client-body isolation, static export authorization, detached model roots with links to request spans, and the real phase-span noise filter. Its 12-model-span check still passed in patched local ARM64 and AMD64 image builds; the prior 1.98.0 pin is rejected. Only the patch and guard RUN --network=none steps claim disabled external networking, not dependency installation for the entire build.

This provider-agnostic guard exercises a synthetic chat-only lifecycle, not provider handlers, streaming, or embedding instrumentation. It is installed-package capability evidence, not database-backed key authentication, live Phoenix ingestion, production E2E, or migration coverage. The receiver uses synthetic credentials and no provider or database. It cannot catch the Ollama embedding defect by itself; the additional guard below exercises that provider path. Successful builds do not authorize deployment across the 1.98.0 → 1.106.4 gap.

Local async-only fix and build guards

services/litellm/patch_ollama_embeddings.py is a local source patch, not an upstream backport. The Dockerfile keeps the immutable 1.106.4 base above. The patch requires that installed version and the exact whole-file SHA-256 for litellm/llms/ollama/completion/handler.py:

• Original: b4699cf66d75c0282cb8fc22b8ffff1fd9ad6a74235e29bd95252c585bf572e3.
• Patched: 45b4147265096328520271c5ee0bdb7ae715c221c4ff2374d4c31fd8e696a015.

Version/hash, AST and transport-shape checks fail closed before a same-directory atomic write. Only ollama_aembeddings gains logging_obj.pre_call and post_call around transport; removing the two hooks must restore the entire original AST. Synchronous ollama_embeddings is untouched: upstream OTel v2 sync SDK success/failure callbacks are no-ops and not this patch's scope. A sync pre-call could leave an open span. Synchronous HTTP clients through the proxy still use the fixed async server route, not the synchronous LiteLLM SDK path.

services/litellm/check_ollama_embedding_tracing.py runs real aembedding, the proxy auth-metadata adapter and Phoenix callback/exporter against loopback mocks for Ollama /api/embed and /api/show and OTLP. Two batch successes route to two projects, two provider failures (HTTP error and invalid JSON) emit ERROR spans, and two pre-transport parameter rejections emit no phantom spans. The guard requires exactly four model spans, preserved request/response semantics and no leaked carriers; it does not manually invoke logging hooks or inject instrumentation state. Unpatched 1.106.4 failed with zero spans. Both patched AMD64 and ARM64 builds passed this guard and the existing 12-span chat guard. This adapter-level check is not database-backed auth, full HTTP proxy or live provider/Phoenix acceptance; the DB rehearsal below supplies separate evidence.

The Docker context allowlist contains only Dockerfile, Dockerfile.dockerignore, otel_quiet.py, patch_ollama_embeddings.py, check_phoenix_routing.py, check_ollama_embedding_tracing.py and config.yaml. The patch and both guards are also private HMAC deployment inputs, alongside the Dockerfile/context rules and existing runtime inputs. Keep those lists aligned; secrets, docs and rehearsal evidence do not enter the image context. On source-pin upgrades, review/remove the local patch and update all applicable version/hash gates, image-test pins and fixtures together. Never disable a failing gate to upgrade; rebuild both architectures and repeat isolated acceptance before requesting rollout approval.

Initial rehearsal: historical failure

The final run of the original rehearsal was pr507-d8b9730900, 2026-10-26 09:26:17–09:28:25 UTC, for draft PR #507 (source commit d839f37). The aggregate report records exact image/source digests and bounded results, without database rows, raw logs, credentials, response IDs, or collector span inventories.

Snapshot and isolation. One production read-only pg_dump at 09:05 UTC supplied the custom-format snapshot reused throughout, including this final run and rollback; the final run made no production calls. The dump was 9,029,830 bytes, SHA-256 4459352c3d4f3738a609ab6a8341ff727f4fde439f53aa235d547b3a3faa6aa4. The isolated database matched production PostgreSQL 16.15 Alpine, linux/amd64, en_US.utf8 collation/ctype and UTF8 encoding, using postgres@sha256:f4610795587492cd4fb887c2dc3de301b813278bb6f616992061141b5f15ac36. The exact PR wrapper was built for AMD64 as sha256:d1b99b920869deeafe8b0bedde3f5566342e9c7d5ef5e41c6d434a511c140854, on the 1.106.4 base digest above.

All containers shared the database's --network none namespace: only lo was UP, with zero IPv4 routes and zero published ports. Kernel-created tunnel interfaces existed but were inactive, and containers had no CAP_NET_ADMIN. There was no production database migration, provider/model call, key update, or trace cleanup. The snapshot export was the only production database operation.

Database migration passed. The migration-only CLI (--skip_server_startup) exited 0 in 20.73 seconds. The restored database had 75 base tables, including _prisma_migrations. All 74 original application-table counts and original-column fingerprints were unchanged, using only the reviewed LiteLLM_ShadowEvalJob.api_key_id → target_id comparison alias from 20260831000000_shadow_eval_typed_targets; there were no other waivers. That table had zero rows, so this does not prove a populated rename. The migration ledger grew 157 → 213 (56 added), and all 213 checksums matched the exact image. There were zero invalid indexes, zero unvalidated constraints, and two request-log indexes.

Synthetic runtime passed functionally, but trace acceptance failed. Runtime used that same isolated migrated schema only after truncating all 97 application tables and verifying zero rows; production data was never exposed to proxy runtime. Only synthetic keys/budgets, local mock providers, and a local OTLP receiver were used. Database-backed key creation, metadata, model allowlists, invalid-key 401, unauthorized-model rejection, normal chat for two projects, streaming, OpenAI and Ollama embeddings (768 dimensions each), and ordered fallback (three failed primary attempts, one successful secondary) passed. Chat, stream, fallback, and OpenAI embedding model spans reached the correct projects. The sole failure was no companion-archive-embedding Ollama model span within 15 seconds, despite a successful embedding response.

Source inspection confirmed unpatched ollama_aembeddings in llms/ollama/completion/handler.py omitted logging_obj.pre_call and post_call. OTel v2 _take_completed_call drops a completion when the carrier is None and there is no api_call_start_time. This was an instrumentation failure, not a restore/migration failure; the local async-only patch now addresses it. Do not weaken the test or claim the upstream upgrade alone fixes Ollama tracing. Earlier fixture corrections (OpenAI-only encoding_format sent to Ollama, route-phase classification, alias/model matching, and legacy admin metadata) were separate from the missing-hook defect.

Rollback passed within a bounded scope. A fresh restore of the same pre-upgrade snapshot, not a downgrade of the migrated schema, exactly matched the baseline schema/count/digest fingerprint including the migration ledger. After scrubbing and verifying zero rows in all 74 application tables, the old 1.98.0 AMD64 image passed synthetic auth, chat, streaming, both embedding providers, and fallback checks. Its /key/generate admin API rejects Phoenix project fields, so legacy tests used ordinary metadata and no routing assertions. These are not proof that original production keys work, that production Phoenix ingests traces, or that production E2E succeeds.

Historical cleanup and decision. All initial rehearsal containers and its temporary raw dump were removed after evidence recording (no secure-erasure claim). At that point the missing Ollama span blocked acceptance. That failed report is preserved, not overwritten by the successful patched repeat below.

Patched rehearsal: isolated acceptance passed

Final run pr507-fixed-084c08e6a6 started 2026-10-26T10:03:22.894553Z and finished 2026-10-26T10:06:15.741139Z, using source 9623e82. The new aggregate report binds the parent result, baseline and fixture hashes to this source and exact images. It publishes no rows, credentials, raw logs, detailed response/trace/span IDs or collector inventories.

Fresh snapshot and exact images. After the earlier dump was removed, a new production read-only pg_dump over SSH supplied this run and its rollback. The custom-format dump was 9,029,830 bytes, SHA-256 ba8bd18a46e1826386091b5c4095e18dc2976ab6bd44227ec08e44c83a0c5a34. PostgreSQL 16.15 Alpine AMD64 matched the production source image; UTF8 encoding and en_US.utf8 collation/ctype were independently verified.

Artifact	Exact image ID or source digest
Patched AMD64 wrapper, used for DB rehearsal	sha256:6bfa2fa5c4ca77d3afe16b589ee799c8edeb9037a720d73563f225d180f77f89
Patched ARM64 wrapper, build guards only	sha256:5f89e01377cc76df272365f52554a12b7ab4ebd2d8133d1c53eedfe62348e8ce
PostgreSQL source image	postgres@sha256:f4610795587492cd4fb887c2dc3de301b813278bb6f616992061141b5f15ac36

All containers shared the DB's --network none namespace with only lo UP, zero IPv4 routes, zero published ports and no CAP_NET_ADMIN; inactive kernel tunnel interfaces were not egress. The dump was the only production operation: no deployment, database migration, key apply, provider/model calls or trace cleanup.

Migration passed. The migration-only CLI exited 0 in 23.4 seconds. The restore contained 75 base tables including the ledger. Counts and original-column fingerprints of all 74 original application tables were preserved using only the reviewed LiteLLM_ShadowEvalJob.api_key_id → target_id comparison alias. That table remained empty in the snapshot; populated-rename behavior is still unproven. The ledger grew 157 → 213 (56 migrations), with all 213 checksums valid, zero invalid indexes, zero unvalidated constraints and two request-log indexes.

Synthetic runtime and strict routing passed. All 97 migrated application tables were scrubbed and verified zero before proxy runtime. Only synthetic keys/budgets, local mock providers and the local collector were used. Database-backed key creation/auth, finite budgets, trusted metadata, model allowlists, invalid-key 401, unauthorized-model rejection, two-project chat, streaming, 768-dimensional OpenAI and Ollama embeddings, and ordered fallback (three primary failures, one secondary success) passed.

Strict routing now passed with 7 model spans, 47 allowed gateway/noise spans, and zero model spans in fallback litellm. In particular, the OpenAI and Ollama embedding models both routed to test-b. Complete successful chat response IDs matched the expected projects (including streaming and fallback cases); detailed IDs and raw collector data stay unpublished. A successful embedding response alone remains insufficient: strict model-span acceptance was retained, not waived.

Fresh-restore rollback passed. The same pre-upgrade snapshot was freshly restored to the exact baseline schema/count/digest/ledger state, not downgraded from the migrated schema. After all 74 application tables were scrubbed and verified zero, old 1.98.0 AMD64 passed synthetic auth, allowlists, ordinary metadata, chat/stream, both embeddings and fallback checks. Legacy Phoenix routing was intentionally not asserted: the old admin API bans Phoenix metadata fields, and rollback retains its known routing limitation.

Cleanup, limits and decision. All temporary containers were removed, with no matches in the label-filtered docker ps check. The fixed-run raw dump was also removed; deletion is not a secure-erasure claim. At that point the isolated async fix was verified, not production E2E: original production keys, real providers/current clients and live Phoenix ingestion were unproven. The decision was to keep production unchanged pending review and owner approval for rollout, metadata changes and separate production acceptance. No such actions were performed by this rehearsal or documentation update. The later production result is separate bounded evidence.

Required approval gates and order

These are the retained safety gates for an approved rollout, not instructions to replay the completed migration or permission for another deployment. The 2026-10-26 outcome and its separate evidence are above. For future changes, reassess the starting version/schema, backup point and scope. /opt/homelab is at main 9240794 but still dirty; review it without resetting user changes, then refresh/check inputs and verify canonical-path convergence before deployment. Do not assume it is interchangeable with the serving pinned release merely because Git HEAD matches.

1. Back up and restore-test before upgrading. Take a consistent LiteLLM PostgreSQL backup and retain its matching pre-upgrade image, config.yaml, Compose configuration, and securely recoverable secrets. Protect backup access as carefully as live credentials. Restore the backup in isolation and verify keys, team/key metadata, budgets, and configuration. Provider usage records cannot regenerate virtual keys or routing/configuration. Follow the current LiteLLM recovery procedure and logical-backup contract: services/litellm/backup.yml protects validated logical generations under /var/backups/postgres-logical/litellm, not hot PGDATA. Its producer defaults to an offline plan; export and restore verification need separate explicit approval. A manifest or database-only import smoke test is not full application recovery or migration-rehearsal evidence.
2. Rehearse 1.98.0 → 1.106.4 against the isolated restored database. Proxy startup may run schema migrations. Disable provider and external network egress, including production Phoenix export, before starting the restored stack; allow only isolated rehearsal dependencies. Never point rehearsal at the live database or live provider endpoints. Exercise startup/migrations, database-backed virtual-key authentication and authorization, metadata preservation, and representative current clients' streaming, embeddings, and provider fallback behavior against local test backends. The image guard is not a substitute. Any subsequent live-provider/client tests need separate approval for cost, content retention, and egress. Record results and rehearse the matching old-image/database/config restore before scheduling production. The patched run above satisfies the bounded synthetic isolated gate for source 9623e82 and its exact AMD64 wrapper, including strict Ollama span acceptance. It does not validate live providers/current clients, original production keys or Phoenix-server ingestion. Repeat after relevant source, image or schema drift; keep strict trace checks enabled.
3. Approve and schedule the maintenance window and rollback point. For any future attempt, confirm the window and rollback point within explicit owner authorization; the completed rollout does not waive those gates. Coordinate client traffic and database writes, take a fresh consistent pre-upgrade backup, and agree on how any post-upgrade writes would be handled if restoration is needed. Confirm the backup, restore evidence, old image/config, and acceptance criteria before authorizing deployment. Do not use a normal deploy/restart as a rehearsal.
4. Explicitly build and recreate the approved proxy image. The merged deployment-input contract tracks Dockerfile, Dockerfile.dockerignore, otel_quiet.py, patch_ollama_embeddings.py, both check_phoenix_routing.py and check_ollama_embedding_tracing.py, config.yaml, Compose, secret/rendering and shared-engine inputs with a private HMAC-SHA256 fingerprint. The homelab Repo on_pull refreshes inputs before deploy-infra-services conditionally selects changed stacks. LiteLLM registers .deploy-inputs/fingerprint for Redeploy, enables run_build = true, and uses a read-only predeploy --check. The explicit deploy-litellm force/retry Procedure pulls homelab and refreshes inputs before unconditional DeployStack; it bypasses change detection, not input verification or maintenance/data gates. A fingerprint records desired inputs, not successful build, deployment or acceptance. Any future rollout requires explicit approval after steps 1–3; this documentation authorizes no additional deployment.

Ensure the approved checkout is present on the host and follow the credential-preserving rendering contract, not Setup's full-stack start. LiteLLM retains required POSTGRES_PASSWORD and LITELLM_MASTER_KEY, opting in only with --literal POSTGRES_PASSWORD and --url-encode POSTGRES_PASSWORD=POSTGRES_PASSWORD_URLENCODED. The database gets the unchanged raw password; the proxy URL requires the generated URL-encoded copy. Regenerate old .env files from SOPS before manual config/build/recreate; missing derived values fail closed, and a derived key already in SOPS is refused. This is a representation change, not credential rotation. Before live cutover, privately verify exact byte/string equivalence between the existing working credential, SOPS, Compose's raw value and the decoded URL password, and verify database authentication with that same credential. Stop on any mismatch or auth failure; do not expose values in logs or evidence. Compose retains pull_policy: build; non-secret config.yaml is baked into the image at /app/config.yaml, not bind-mounted. Keep credentials in SOPS-rendered runtime environment references, never literal image config. For a future approved manual rollout, only after reviewing/reconciling the dirty shared checkout without resetting user changes, refresh and verify inputs at the canonical path using the same authorized root identity as rendering. Its current Git HEAD alone is not deployment-input convergence:

sudo -n bash /opt/homelab/scripts/komodo-deploy-inputs.sh litellm
sudo -n bash /opt/homelab/scripts/komodo-deploy-inputs.sh litellm --check

Stop on refresh/check failure; neither command builds or deploys. From /opt/homelab/services/litellm, the approved rollout command is sudo -n docker compose up -d --build --force-recreate --no-deps litellm. This builds the guarded image and recreates only the proxy, not the database; it does not refresh inputs or reconcile metadata. A restart or metadata change alone cannot install changed image inputs, including config. See the image upgrade instructions. Verify the deployed version/build and health without printing environments or credentials. Hold ordinary deploy-infra / deploy-infra-services and explicit deploy-litellm execution until the backup, rehearsal, and maintenance approvals are complete. 5. Only after the approved image rollout, apply the reviewed metadata map. Review the offline preview and live read-only check, then explicitly opt in from the repository root with the admin credential securely injected (commands below). A pre-apply drift result is not permission to skip review. Resolve team conflicts deliberately; do not overwrite unrelated metadata. The updater is non-transactional, so inspect partial state after failure. No historical Phoenix projects/traces are renamed or deleted, and deploy does not run this reconciliation automatically. 6. Run approved acceptance checks. Use the opt-in verifier above with at least two distinct virtual keys/projects and explicitly selected allowed chat models. Require exact response-ID attribution across all Phoenix projects, not merely a successful completion or any recent trace. Separately test current clients' auth, streaming, embeddings, and fallback behavior; the verifier's non-streaming chat cases do not cover these. Record bounded routing evidence and client results before declaring remediation complete.

Manual metadata commands for step 5 only, after approval and image rollout. On infra-services, from the reviewed, approved checkout (current HEAD alone does not approve dirty /opt/homelab), copy only the non-secret helper/map into the running container. Its Python environment already has PyYAML and the runtime master key; do not extract or print the key. These commands use the internal loopback API and do not expose administrative routes through Traefik:

python3 scripts/configure-litellm-phoenix-projects.py  # offline preview
# Copy prepared code, not credentials; no restart required.
docker cp scripts/configure-litellm-phoenix-projects.py litellm:/tmp/configure-phoenix-projects.py
docker cp services/litellm/virtual-keys/phoenix-projects.yml litellm:/tmp/phoenix-projects.yml
docker exec litellm /app/.venv/bin/python /tmp/configure-phoenix-projects.py \
  --config /tmp/phoenix-projects.yml --base-url http://127.0.0.1:4000 --check
# Expected pre-apply drift is nonzero. Review it before this explicit write:
docker exec litellm /app/.venv/bin/python /tmp/configure-phoenix-projects.py \
  --config /tmp/phoenix-projects.yml --base-url http://127.0.0.1:4000 --apply
docker exec litellm /app/.venv/bin/python /tmp/configure-phoenix-projects.py \
  --config /tmp/phoenix-projects.yml --base-url http://127.0.0.1:4000 --check
docker exec litellm rm -f /tmp/configure-phoenix-projects.py /tmp/phoenix-projects.yml

Rollback

The production rollout retained the actual old wrapper, image archive and private configuration in R/rollback, plus the verified pre-migration logical generation listed above. No production restore was run. Do not treat dirty /opt/homelab, despite its current HEAD, as verified recovery configuration; the migrated database must not be opened by the old image without a separately approved matching restore.

Retain the prior base image from Dockerfile Git history or PREVIOUS_IMAGE in tests/test_litellm_phoenix_image.py:

ghcr.io/berriai/litellm:main-stable@sha256:20b5044b619055374061a6d5b7b08754cad75aeabbf82ddf4f69cc0cf80ddaf4

Retain the matching previously built wrapper image/config as well; the base image alone does not include local additions. If rollback is required, stop proxy writes and restore the old image plus its matching pre-upgrade database and configuration/secrets using the rehearsed procedure. An old image against a migrated database is not a safe rollback. Do not attempt an unverified schema downgrade or delete database volumes (docker compose down -v is not recovery). Restoration returns to the backup's recovery point, so reconcile later writes under the approved plan. Returning to 1.98.0 also returns to its known per-key routing limitation; it does not preserve the remediation.

Troubleshooting

Symptom	Likely cause	Check
/v1/models returns 401	Missing/wrong LiteLLM key	Use a virtual key or LITELLM_MASTER_KEY on infra-services
/v1/models works, chat fails	Backend/provider unavailable	For ollama-llama3.1, confirm the workstation Ollama service is running
Phoenix /v1/projects returns 401/403	Missing/wrong Phoenix API key	Generate a Phoenix System API Key and put it in LiteLLM .env as PHOENIX_API_KEY
Chat works, Phoenix has no traces	Callback/env mismatch or collector issue	Confirm callbacks: ["prometheus", "arize_phoenix"], PHOENIX_COLLECTOR_HTTP_ENDPOINT, PHOENIX_PROJECT_NAME, and PHOENIX_API_KEY
Phoenix UI works but OTLP fails	Wrong collector protocol/path or missing authentication	Authenticated OTLP/HTTP is http://192.168.6.124:6006/v1/traces; authenticated OTLP/gRPC is http://192.168.6.124:4317, with no /v1/traces path
Model calls still land in litellm	Old LiteLLM implementation, metadata drift, or unapproved/unapplied map	Check the running version, mapper --check, and approved rollout status; 1.98.0 ignores per-key projects even with OTel v2 enabled
Correct model project but separate traces	Intentional routed-root detachment	Follow span links; matching application project names do not automatically join trace trees

LiteLLM may not log every Phoenix export failure loudly. If the chat succeeds but Phoenix has no trace, use exact response-ID evidence from the verifier rather than quiet LiteLLM logs or an arbitrary recent trace.

Add Models

Edit the non-secret services/litellm/config.yaml and rebuild/redeploy the proxy through the approved deployment-input flow above. The config is baked into the image; editing the host file or restarting the proxy does not update it:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

Provider API keys live in .env from SOPS. After Phase 9 Ollama returns as an always-on guest, add a second model entry pointing at that host on the Servers VLAN instead of the workstation backend.

ComfyUI Ollama (workstation direct traces)

ComfyUI on the Personal-VLAN workstation sends direct OTLP traces from OllamaGenerateV2 / OllamaChat into Phoenix project comfyui-ollama (separate instrumentation from LiteLLM; after the approved routing rollout, captain_comfyui model spans target this same project, while gateway operational spans remain in litellm). Sharing the project does not automatically join traces.

Item	Value
Fork	spadoople/comfyui-ollama
Upstream	stavsap/comfyui-ollama
Install path	F:\ComfyUI\custom_nodes\comfyui-ollama (WSL: /mnt/f/ComfyUI/custom_nodes/comfyui-ollama)
Phoenix project	comfyui-ollama
Node docs	PHOENIX.md

Install or refresh via git clone — not ComfyUI Manager (Manager updates overwrite customizations):

# From homelab repo root in WSL
bash scripts/workstation/install-comfyui-ollama.sh

Merge upstream releases:

bash scripts/workstation/merge-comfyui-ollama-upstream.sh

Set before starting ComfyUI. Inject PHOENIX_API_KEY securely with a Phoenix System API key; the collector requires authentication even on the LAN. Do not paste the key into shell history:

export PHOENIX_ENABLED=1
export PHOENIX_PROJECT_NAME=comfyui-ollama
export PHOENIX_COLLECTOR_ENDPOINT=http://192.168.6.124:4317
export PHOENIX_CAPTURE_CONTENT=1
export PHOENIX_BATCH=0

Expected in Phoenix after one generation: span kind LLM, non-zero token counts, readable input/output messages.

Direct Phoenix Instrumentation

Use OpenInference/application instrumentation for retrieval, tools, agent steps, and custom workflow spans, including applications whose model calls go through LiteLLM. Gateway instrumentation alone cannot observe those steps, and routed model roots are linked rather than automatically parented into application traces.

From an authorized host on 192.168.6.0/24, the authenticated gRPC OTLP endpoint is (no /v1/traces suffix; that path belongs to OTLP/HTTP on port 6006):

http://192.168.6.124:4317

Example (requires securely injected PHOENIX_API_KEY and provider credentials; running the model call can incur cost and retain prompt/completion content):

pip install openinference-instrumentation-openai arize-phoenix-otel openai

import os

from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor
import openai

register(
    endpoint="http://192.168.6.124:4317",
    protocol="grpc",
    project_name="homelab",
    headers={"authorization": "Bearer " + os.environ["PHOENIX_API_KEY"]},
)
OpenAIInstrumentor().instrument()
client = openai.OpenAI()
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "ping"}],
)

Edge Auth

Path	Authentik	Auth mechanism
litellm.infra.realemail.app/ui	Yes	Authentik forward-auth
litellm.infra.realemail.app/v1	No	LiteLLM virtual key
phoenix.infra.realemail.app	Yes	Authentik forward-auth
192.168.6.124:4317 (OTLP/gRPC)	N/A	Phoenix API key in authorization metadata; LAN reachability alone is insufficient
192.168.6.124:6006/v1/traces (OTLP/HTTP)	N/A	Phoenix API key in Authorization: Bearer …

Configure Authentik proxy providers for litellm.infra.realemail.app and phoenix.infra.realemail.app per authentik-cross-host-sso.md.

Cloudflare DNS: both hostnames are covered by *.infra.realemail.app -> 192.168.6.17.

Operations

These are deployment commands, not preparation checks or rollback instructions. The pinned rollout is already serving from R/source/services/litellm. /opt/homelab HEAD is now main 9240794, with unchanged dirty-status output; file-content identity was not verified. Current HEAD alone does not make the canonical Komodo path /opt/homelab/services/litellm deployment-ready. Review user changes without a hard reset, refresh/check inputs and verify path convergence, then obtain approval for any future deployment under the rollout gates. Follow the existing LiteLLM rendering contract, not Setup's initial full-stack start. LiteLLM on infra-services requires a build for image/guard/config changes; config.yaml is baked into the image, not bind-mounted. After those future gates only:

cd /opt/homelab/services/litellm || exit 1
sudo -n docker compose up -d --build litellm
sudo -n docker compose logs -f litellm

Phoenix on prox LXC 124 (192.168.6.124) — confirm access before running commands:

• Human access: the host access table lists an operator route, but confirm the intended account, authorized key, SSH host identity, any jump host, and Docker privileges with the owner before using it. Ansible's ansible_user: root is an automation setting, not proof of the human principal or proof that a separate someone account is absent.
• C&C access: use the established infra-services inbound procedure for the appropriate human or agent identity, then the controller-to-guest access guidance. The configured patching route is from infra-services to root@192.168.6.124 using the root-readable patch-controller key documented in coordinated OS patching. This is a separate privileged route requiring owner authorization, not a default workstation login. Keep the controller key on infra-services; do not copy it to a client.
• Stop if access is unconfirmed: ask the owner to establish the principal, key, jump path and required privileges. Do not guess an alias, substitute root for a human account, enroll keys or bypass SSH host-key checking to proceed. These source references do not verify a live login.

After the approved session is established on Phoenix, not on the workstation or controller:

cd /opt/homelab/services/phoenix || exit 1
sudo -n docker compose up -d

Both stacks are tier 2. Postgres volumes are documented in each backup.yml.

Related

• LiteLLM service README
• Phoenix service README
• phoenix host
• comfyui-ollama fork — workstation ComfyUI tracing
• ollama host - retired; Phase 9 inference uses a new guest