Zed sessions and canonical observations

State and scope

Deployed; synthetic Phoenix Session ingestion passed. Two fresh native Zed threads and UI acceptance remain pending. The owner confirmed the existing native routing/payload path, merged PR #574 and approved continuing the scoped rollout. Revision ee2385ca was built and only the existing proxy recreated at 2026-11-02 05:12:33 UTC. This adds telemetry grouping/filtering only; no new credential or Zed setting is required. Future rollouts retain separate approval.

No Zed settings, SOPS values, virtual-key policies, shared callbacks, retention, request parameters, upstream cache keys, version pins or cloud destinations change. See Zed observability for the existing auth/privacy gates, limits and recovery contract. Session metadata is derived after those gates; it can never enable content capture or bypass redaction.

Session attribution

For each authorized content observation, inspect only prompt_cache_key copies in top-level logging kwargs, optional_params and the proxy's request-body snapshot. All supplied copies must be the same canonical lowercase, nonnil UUID. SDK-normalized None defaults are ignored; explicit raw JSON null invalidates attribution. Values are length-bounded before parsing and are never coerced.

A valid value becomes this span attribute:

session.id = homelab:zed:thread:<UUID>

The installation and project names are part of the ID. Phoenix 20.26.1 session IDs are globally unique, not scoped by project: reusing an ID in another project can attach traces to its existing session. The namespace is not a fallback ID; missing/invalid/conflicting cache keys leave session.id absent while otherwise- authorized content capture continues. Never replace it with one constant header, traceparent, key alias or a time-window grouping.

Each observation keeps an independent root trace. Phoenix's normal ingestion creates/attaches Sessions from session.id; no separate session-creation API or SQL is required. The live smoke created two explicitly synthetic validation Sessions; no historical records were regrouped or deleted.

Attribute	Meaning
zed.session.source	prompt_cache_key: the attempted attribution mechanism, including when the value is absent
zed.session.attribution	thread_cache_key, missing, invalid or conflict
session.id	Present only for a validated, agreeing UUID, with installation/project namespace
zed.observation.kind	content: this callback's canonical content observation, not the shared gateway observer
zed.event_kind	Existing success or failure logical observation

Malformed/rejected cache-key text is not promoted into telemetry attributes. Existing full-content capture is still not a secret scrubber for message/tool text. Session IDs make conversations easier to correlate; they are linkable identifiers, not authentication or authorization boundaries.

What the Zed identifier means

Installed Zed 1.26.2 was checked against its tagged request builder and OpenAI converter:

• Normal native Agent calls use their owning thread ID as the wire cache key; later turns, tool rounds and resumed requests ordinarily retain it.
• Thread-title/summary requests can use the same owning ID even when they use Luna rather than Sol, so those model calls can group together.
• Git commit generation can omit the key and remains ungrouped. Do not infer purpose merely from the Luna model name.
• A subagent has a separate child-thread ID. Its Session remains separate; cache keys contain no structured parent relationship.
• Native compaction can preserve the thread/cache ID, but grouping subsequent calls does not create compaction lifecycle or local-tool timing spans.

The converter prefers an explicit cache key over the thread ID. Consequently, this is cache-key-derived correlation, not authenticated native thread identity. Reusing an explicit UUID cache key can merge logical conversations; changing it can split one. Future Zed versions or other clients using the key must be requalified. No identifier is guessed from headers, arbitrary metadata, model, prompt text, response IDs or temporal proximity.

The offline guard proves these fields survive the installed LiteLLM SDK and provider transformation. It does not establish the keys actually sent by two live Zed threads; that remains the synthetic post-deploy acceptance below.

Duplicate-safe filtering

The unchanged global Phoenix callback and the dedicated content callback can observe the same model call. Do not sum all project spans as unique Zed calls. Only the content callback adds the Session and canonical label. For newly emitted spans, use these equivalent span-selection criteria:

attributes["zed.observation.kind"] == "content"

For completed logical-call counts/tokens/cost, also require:

attributes["zed.observation.kind"] == "content" and attributes["zed.event_kind"] == "success"

Inspect failures separately using zed.event_kind = failure; do not discard them from diagnostic review. A failed attempt and later success can both exist under one LiteLLM call ID. Deduplication is bounded/process-local by (call_id, event_kind): these observations are not a complete provider-attempt or billing ledger. LiteLLM cost is not authoritative OpenAI billing.

No persisted Phoenix dashboard/view is provisioned. Apply the canonical filter where supported; an unfiltered project dashboard may still count both observers. The Session feature itself only attributes the new content observations. It does not suppress, rewrite or move the global callback's observations or historical data.

Older content spans lack the new canonical label and Session. Their unchanged name, zed.content.completed, can identify historical content observations for explicit analysis, but does not retrospectively group them. No historical migration, replay, cache-key recovery or trace deletion is proposed.

Offline verification

The existing image build guard now expects 17 synthetic spans, retaining all prior auth/privacy/tool/usage/streaming checks and the 120s/60s finite deadlines. It exercises real proxy stamping, SDK Router/Responses entrypoints, provider request transformation, callback dispatch, SDK serialization and a loopback OTLP receiver. Only provider HTTPX transport is replaced with fixtures.

Session cases cover the Sol function-call/result round trip on UUID A, a Luna helper on A, a separate Luna call on UUID B, and a no-key Luna call. Provider-wire keys remain unchanged; final Logging copies agree; A shares a namespaced Session, B differs, missing keys remain absent, and each observation has a distinct trace ID. Other cases test malformed/nil/oversized/conflicting copies, namespace changes, forged metadata/headers, redaction, unchanged input objects and gate precedence. These are fixture identities, not executions of Zed or real subagents.

Run tests/test_zed_phoenix.py in an already-local candidate image with --pull never --network none --read-only, bounded /tmp tmpfs and read-only source mounts. When testing mounted source against an older image, use a working directory such as /tmp and explicit PYTHONPATH=/capture; /app as cwd can shadow the mounted module with a baked older callback. A new packaged-image test must instead import its actual baked callback. Do not install packages or pull a mutable image as a substitute for testing the canonical Dockerfile.

Deployment and synthetic ingestion result

The approved rollout held the relay lock, checked an idle stack/no pending intent, retained a fresh validated logical backup and root-private rollback inputs, and fast-forwarded without resetting/archiving user work. Scoped inputs were rendered, refreshed and checked before the native build. The network-disabled image guard passed 17 spans; the built callback/config hashes matched reviewed source and every Python package version matched the previous image.

The proxy started at 2026-11-02T05:12:33.397669539Z, using image sha256:a288ab76601ec3531fa5f06f60b67c22bfdf9455f51996f219affa4a3deb27d5. Only it was recreated. Readiness, package/code identity, complete runtime env, capture gates, stable key policies and unchanged DB container were verified. Rollback inputs and safe status/build/smoke evidence are retained root-private at /var/lib/homelab/rollouts/litellm/zed-sessions-ee2385ca.

A separate-process callback-to-Phoenix smoke, using only synthetic input, output, cache keys and usage, verified:

• Six canonical content spans, six distinct root trace IDs, correct namespace and model attribution; no mutation of input objects.
• Session A has three traces: two Sol and a Luna helper. B has one Sol trace and a different backend Session record. Fixture token totals aggregated as expected.
• No-key Luna and invalid-key Sol spans remained ungrouped but retained content.
• Duplicate and non-Zed gates passed; no paid test model request was made.
• A Python socket guard reported zero denied external attempts, not a complete packet/egress audit. Only loopback admin and existing LAN Phoenix were used.

The supported Session GET API, not only span attributes, confirmed grouping. A later metadata-only sample found 20 recent non-synthetic Sol content spans with valid attribution across four Session IDs. No payloads were inspected or raw IDs printed; mapping known UI threads/associated Luna calls still needs the owner test. A first read-only schema probe exceeded its 10-second bound; a bounded 25-second read succeeded. No execute operation was retried and no guard was bypassed. This smoke did not authenticate a raw virtual key, run Zed, execute tools or qualify actual provider cache/billing behavior. The two synthetic Sessions remain visible; do not expect the project to contain only your new test Sessions.

Approval-gated rollout and acceptance

The deployment above completes steps 1–3 for this revision. The next owner action is step 4, using two brand-new threads so their Luna title/summary calls also occur after rollout. No credentials need to be re-entered and no further deployment is required for that test.

1. Review/merge the focused source PR and its exact-head checks. Reconfirm the deployed Zed key identity, source/image, scope, rollback inputs and recoverable backup without printing credentials or actual cache/thread IDs.
2. Obtain approval for a proxy-only rebuild/recreation using the existing deployment contract. Both changed Python files are already fingerprinted/allowlisted image inputs. Refresh/check them, require the offline guards, and build the new image. A restart or the earlier environment-only activation's --no-build command does not install this callback change. No version/DB/schema upgrade.
3. Preserve current SOPS enable/hash, Zed marker/model grants and all other clients' policies. There are no new settings or credentials for Sessions. Verify the new image, readiness and unchanged DB/global capture after the approved cutover.
4. Use two new fixture-only native Zed threads. Send at least two turns in A; verify its Sol calls and any associated Luna title/summary group in one Session. Start B close in time with the same model; verify it gets a different Session. Inspect equality/separation locally rather than posting raw IDs/transcripts.
5. Where applicable, test a no-key helper/commit fixture and a synthetic subagent: missing attribution remains visible but ungrouped; a child uses its own Session, with no fabricated parent link. Resume/compaction testing must distinguish stable grouping from independently unobserved lifecycle events.
6. Compare provider requests and canonical-filtered totals with the previous path. Confirm cache keys, model parameters and content policy unchanged; validate actual Phoenix Session list/trace presentation. JSON payload rendering and Sessions grouping do not guarantee a fully rendered native conversation panel. Record these live results before calling Sessions accepted.

Rollback uses the matching pre-change image/config under separate restart approval; no new secret or DB schema needs undoing. Old traces/Sessions stay intact. Do not remove user work or delete telemetry to make grouping look consistent. Existing capture gates can disable the optional content path if necessary; see the main rollback procedure.