LLM Observability - LiteLLM + Phoenix¶

This stack has two jobs:

Tool	Host	URL	Daily job
LiteLLM	infra-services	`https://litellm.infra.realemail.app`	One OpenAI-compatible gateway, virtual keys, budgets, usage, Prometheus metrics
Phoenix	prox LXC 124	`https://phoenix.infra.realemail.app`	Trace inspection, prompt/RAG debugging, evals, datasets, experiments

Use LiteLLM for normal traffic. Open Phoenix when a request behaved strangely, you are tuning a prompt chain, or you need evidence about what happened inside a multi-step LLM workflow.

Mental Model¶

Most clients should only know about LiteLLM:

client -> https://litellm.infra.realemail.app/v1 -> provider/backend
                                              |
                                              +-> Prometheus metrics
                                              +-> Phoenix traces

Phoenix is not another gateway. It is the trace workbench behind the gateway. When LiteLLM tracing is healthy, a request sent to /v1/chat/completions should show up as a recent trace in Phoenix.

Day In The Life¶

One-time setup for a new tool¶

Open the LiteLLM Admin UI at https://litellm.infra.realemail.app/ui.
Create a virtual key for the client, such as comfyui, cursor, or homelab-script.
Give the key a budget/rate limit that matches the blast radius of that tool.
Configure the client:

Base URL: https://litellm.infra.realemail.app/v1
API key:  <LiteLLM virtual key>
Model:    one of the names from /v1/models, for example ollama-llama3.1

The Authentik browser session only protects the UI. API clients authenticate with LiteLLM virtual keys.

Normal usage¶

Use LiteLLM as the base URL for OpenAI-compatible tools.
Check LiteLLM UI for per-key spend, model usage, and key management.
Check Grafana -> Homelab -> LiteLLM for request rate, token throughput, error rate, and cumulative spend.
Use Phoenix only when you need to inspect a run: inputs, outputs, timings, tool/retriever spans, errors, or eval candidates.

When something looks wrong¶

Start in LiteLLM if the problem is "the request failed" or "this key/model is too expensive." Start in Phoenix if the request succeeded but the answer was bad, slow, incomplete, or hard to explain.

For prompt/RAG work, the useful loop is:

Send the request through LiteLLM.
Open Phoenix and inspect the trace tree.
Find the span where quality changed: prompt construction, retrieval, tool call, model call, parser, or post-processing.
Adjust the prompt/code/data and run it again.
Promote good examples into a dataset or eval when repeated testing matters.

E2E Tests¶

There are two useful tests: a quick gateway smoke test and a full trace test.

Quick Gateway Smoke¶

From any machine that has a LiteLLM virtual key:

curl -sS https://litellm.infra.realemail.app/v1/models \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY"

Expected: HTTP 200 and a model list. Current local Ollama models: ollama-llama3.1, ollama-north-mini-code, ollama-qwen3.6-27b-mtp, and ollama-qwen3.6-27b-heretic-mtp.

Then send one completion:

curl -sS https://litellm.infra.realemail.app/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama-llama3.1",
    "messages": [
      { "role": "user", "content": "Reply with exactly pong." }
    ],
    "max_tokens": 20
  }'

Expected: HTTP 200 with a chat completion. If this fails while /v1/models works, the configured backend may be offline. The ollama-* models point at the owner's Personal-VLAN workstation (192.168.3.17:11434), so they only respond while that workstation and Ollama are running.

Full LiteLLM -> Phoenix Trace Test¶

Run this from infra-services. It uses host-local secrets without printing them:

cd /opt/homelab/services/litellm
set -a
. ./.env
set +a

curl -sS -o /tmp/litellm-chat.json -w "chat_http=%{http_code}\n" \
  https://litellm.infra.realemail.app/v1/chat/completions \
  -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama-llama3.1",
    "messages": [
      { "role": "user", "content": "Homelab trace smoke test. Reply with exactly pong." }
    ],
    "max_tokens": 20
  }'

sleep 5

curl -sS \
  "http://192.168.6.124:6006/v1/projects/litellm/traces?limit=5&sort=start_time&order=desc&include_spans=true" \
  -H "Authorization: Bearer ${PHOENIX_API_KEY}" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print("trace_count=" + str(len(d.get("data", []))))'

Expected:

chat_http=200
trace_count greater than 0
https://phoenix.infra.realemail.app shows a recent trace in the litellm project

Verification note, 2026-07-05: before the callback registration fix in services/litellm/config.yaml, /v1/models, chat completion, and Phoenix /v1/projects all returned 200, but Phoenix returned zero traces. After the callback fix, the full test passed on 2026-07-28: LiteLLM returned HTTP 200 and pong; Phoenix project litellm returned trace 8d077eb51bdf502432daf02bf47d555f with three spans, two LLM spans, all statuses OK, and 64 total tokens. The README Owner TODO is closed.

Troubleshooting¶

Symptom	Likely cause	Check
`/v1/models` returns `401`	Missing/wrong LiteLLM key	Use a virtual key or `LITELLM_MASTER_KEY` on infra-services
`/v1/models` works, chat fails	Backend/provider unavailable	For `ollama-llama3.1`, confirm the workstation Ollama service is running
Phoenix `/v1/projects` returns `401`/`403`	Missing/wrong Phoenix API key	Generate a Phoenix System API Key and put it in LiteLLM `.env` as `PHOENIX_API_KEY`
Chat works, Phoenix has no traces	Callback/env mismatch or collector issue	Confirm `callbacks: ["prometheus", "arize_phoenix"]`, `PHOENIX_COLLECTOR_HTTP_ENDPOINT`, `PHOENIX_PROJECT_NAME`, and `PHOENIX_API_KEY`
Phoenix UI works but OTLP fails	Browser route is not the collector route	LiteLLM writes to `http://192.168.6.124:6006/v1/traces`; direct OTLP gRPC is `192.168.6.124:4317`

LiteLLM may not log every Phoenix export failure loudly. If the chat succeeds but Phoenix has no trace, trust the Phoenix trace query over quiet LiteLLM logs.

Add Models¶

Edit services/litellm/config.yaml and redeploy:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

Provider API keys live in .env from SOPS. After Phase 9 Ollama returns as an always-on guest, add a second model entry pointing at that host on the Servers VLAN instead of the workstation backend.

ComfyUI Ollama (workstation direct traces)¶

ComfyUI on the Personal-VLAN workstation sends direct OTLP traces from OllamaGenerateV2 / OllamaChat into Phoenix project comfyui-ollama (separate from LiteLLM gateway traces in project litellm).

Item	Value
Fork	spadoople/comfyui-ollama
Upstream	stavsap/comfyui-ollama
Install path	`F:\ComfyUI\custom_nodes\comfyui-ollama` (WSL: `/mnt/f/ComfyUI/custom_nodes/comfyui-ollama`)
Phoenix project	`comfyui-ollama`
Node docs	PHOENIX.md

Install or refresh via git clone — not ComfyUI Manager (Manager updates overwrite customizations):

# From homelab repo root in WSL
bash scripts/workstation/install-comfyui-ollama.sh

Merge upstream releases:

bash scripts/workstation/merge-comfyui-ollama-upstream.sh

Set before starting ComfyUI:

export PHOENIX_ENABLED=1
export PHOENIX_PROJECT_NAME=comfyui-ollama
export PHOENIX_COLLECTOR_ENDPOINT=http://192.168.6.124:4317/v1/traces
export PHOENIX_API_KEY=<Phoenix System API key>
export PHOENIX_CAPTURE_CONTENT=1
export PHOENIX_BATCH=0

Expected in Phoenix after one generation: span kind LLM, non-zero token counts, readable input/output messages.

Direct Phoenix Instrumentation¶

Use manual OpenInference instrumentation only for code that does not go through LiteLLM or for experiments where you need custom spans beyond the gateway span.

From any host on 192.168.6.0/24, the gRPC OTLP endpoint is:

http://192.168.6.124:4317

Example:

pip install openinference-instrumentation-openai arize-phoenix-otel openai

from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor
import openai

register(endpoint="http://192.168.6.124:4317/v1/traces", project_name="homelab")
OpenAIInstrumentor().instrument()
client = openai.OpenAI()
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "ping"}],
)

Edge Auth¶

Path	Authentik	Auth mechanism
`litellm.infra.realemail.app/ui`	Yes	Authentik forward-auth
`litellm.infra.realemail.app/v1`	No	LiteLLM virtual key
`phoenix.infra.realemail.app`	Yes	Authentik forward-auth
`192.168.6.124:4317`	N/A	LAN reachability only

Configure Authentik proxy providers for litellm.infra.realemail.app and phoenix.infra.realemail.app per authentik-cross-host-sso.md.

Cloudflare DNS: both hostnames are covered by *.infra.realemail.app -> 192.168.6.17.

Operations¶

LiteLLM on infra-services:

cd /opt/homelab/services/litellm
docker compose up -d
docker compose logs -f litellm

Phoenix on prox LXC 124:

ssh someone@192.168.6.124
cd /opt/homelab/services/phoenix
docker compose up -d

Both stacks are tier 2. Postgres volumes are documented in each backup.yml.

LiteLLM service README
Phoenix service README
phoenix host
comfyui-ollama fork — workstation ComfyUI tracing
ollama host - retired; Phase 9 inference uses a new guest