Zed native-agent observability

State and scope

Implementation merged in PR #569; capture defaults off. Deployment and live acceptance remain unverified. The owner approved one existing LiteLLM instance, a key-gated callback, and overriding Zed's built-in OpenAI base URL. Creating credentials, shared-service deployment/ restart, content retention and paid synthetic inference still require explicit approval. No editor settings or historical chats were changed by the preparation.

Start with the ordered owner to-do list. Only the merge item is complete; do not infer completion of any operational step from it.

Read the gateway runbook and LiteLLM deployment/recovery contract before operational changes. Do not deploy from the owner's dirty working tree. No LiteLLM/Phoenix version upgrade, new gateway, cloud destination or application DB migration is part of this change.

flowchart TD
    Z[Zed built-in OpenAI provider] --> L[Existing LiteLLM gateway]
    L --> O[OpenAI inference using owner API key]
    L --> G[Existing metadata-only Phoenix callback]
    L --> K[Pinned authenticated Zed key plus marker]
    K --> C[Independent bounded content callback]
    G --> P[Local Phoenix]
    C --> P

Evidence and API behavior

Read-only inspection on 2026-11-01 verified:

• Installed Zed 1.26.2, build 20261028.143738; no custom user-data-directory argument. Only model/provider settings were extracted from the user settings.
• Configured default: built-in openai / gpt-6.1-sol, Fast enabled, thinking enabled at medium effort; no language_models override. A saved default does not establish every existing thread's runtime selection.
• Running LiteLLM 1.106.4, OTel SDK 1.45.1, with prometheus and arize_phoenix, exporting to http://192.168.6.124:6006/v1/traces. No global capture flag or payload-printing flag was configured.
• Runtime model inspection found neither Zed model; no Zed-named key or zed Phoenix project existed. Phoenix project API returned 401 without auth and 200 using the existing gateway credential. No historical spans/chats were read.
• Phoenix runtime version/digest, effective retention, telemetry and deployed backup destinations were not independently verified.

The tagged Zed implementation uses native Responses for Sol. Fast requests service_tier: priority; reasoning uses medium effort with API-exposed summaries. Built-in model metadata has a 1,050,000-token context window and 128,000-token output maximum. Do not add model redefinitions or change these budgets/sampling to route traffic.

The default summary/title and commit model is gpt-5.6-luna; subagents ordinarily inherit their parent's selection. An OpenAI-provider URL override covers these features and other requests using that provider. It does not redirect other providers, external agents or edit-prediction services automatically. Explicit per-thread/subagent selections can require a separately reviewed model allowlist.

The two prepared gateway aliases use native OpenAI routing with only the upstream model and an environment-key reference. No Chat Completions translation, fallback, reasoning, cache, Priority or sampling override is configured. Provider acceptance, pricing and any compaction endpoint remain synthetic-live acceptance gates.

Content-isolation contract

LiteLLM 1.106.4's Phoenix capture mode is logger-wide; a separate project/key alone is not selective capture. The callback is registered as zed_phoenix.logger next to, not instead of, the existing callbacks. It does not construct a second LiteLLM OTel logger, set a global tracer provider, change shared logging objects, or turn on global capture. It owns one private SDK provider and one OTLP/protobuf exporter.

Activation requires all of:

1. Runtime ZED_PHOENIX_CONTENT_ENABLED=true (default false).
2. Runtime ZED_PHOENIX_KEY_HASH matching the dedicated virtual key's lowercase SHA-256 hash, not the key itself. Keep it in the private deployment inputs.
3. Nonempty runtime PHOENIX_API_KEY, used only for export authorization.
4. The proxy-resolved key's own metadata containing JSON boolean zed_phoenix_content: true, with matching typed auth object and hash.
5. An allowed model group, gpt-6.1-sol or gpt-5.6-luna, and no effective message-redaction/no-log control. Missing, conflicting or malformed identity evidence fails closed. Client metadata/headers cannot authorize capture.

Requests carrying client x-litellm-call-id are skipped; only proxy-generated call IDs are eligible. The existing global callback still observes Zed requests at its normal content-off policy. A content observation is an additional root span named zed.content.completed, correlated by litellm.call_id and exposed response ID, not an automatically merged trace. Retry failures and subsequent successes can produce separate observations; deduplication is bounded and local to the process. A Phoenix project is not a security/authorization boundary.

Captured detail and deliberate limits

The callback exports bounded, explicit public fields:

• Submitted message context, Responses instructions, full declared tool schemas, assistant output/call arguments, and tool results present in later requests.
• API-exposed reasoning summaries; token details, including cached/reasoning counts when supplied; model/call/response IDs, parameters, route and timing.
• Available LiteLLM response cost (an estimate, not OpenAI's authoritative bill).
• Sanitized failure status/type; raw exception text, stack and provider error bodies are omitted because they may contain secrets.

Headers, arbitrary metadata, auth objects, key material, hidden fields, encrypted reasoning and unsupported/opaque output items are not exported. This is a public field projection, not a raw HTTP archive or a secret scrubber: code/secrets in message text, tool results, schemas and arguments are intentionally captured.

Input/output are JSON attributes; complete tool definitions are in llm.tools. Do not promise Phoenix's native indexed conversation/tool panels: readable JSON and actual UI rendering must be checked in live acceptance. Hidden chain-of-thought cannot be recovered. Only complete successful Responses streams are content-captured; incomplete, failed or interrupted streams may have no custom content observation. The existing gateway error telemetry remains available at its normal policy.

Content limits are 256 KiB per serialized field and 512 KiB total, plus depth/node limits. Oversized fields are omitted whole with explicit *.omitted attributes; JSON is not silently cut into invalid strings. Unsupported items are marked and counted. The full million-token context window is not guaranteed to fit. The SDK queue holds 32 spans, exports batches of four, and permits eight concurrent projections. Queue overflow, shutdown/crash, callback cancellation or export failure can lose data. Fixed-name in-memory counters are available through zed_phoenix.logger.counts.snapshot() in the proxy process; they are not yet Prometheus metrics. Do not treat an empty error counter in another Python process as evidence about the running proxy.

The exporter ignores HTTP proxy/OTLP-header environment settings, refuses redirects, has a two-second socket timeout and no retries, and sends only to the fixed LAN endpoint/project. It has no console exporter. Allow at least 20 seconds for a graceful queued drain; Compose reserves 30 seconds. Atexit cannot complete unsubmitted callback tasks, guarantee a hard deadline or survive SIGKILL.

The LAN OTLP hop is unencrypted HTTP, as in the existing gateway export path; content and the Phoenix bearer credential are visible to a network observer on that hop. Review LAN trust or approve transport hardening before real content. OpenAI still receives submitted inference content; "local traces" does not mean local inference. Zed/SDK/Phoenix's own telemetry and other exporters need separate inspection; this callback is not an egress firewall.

Validation without real content or provider access

The image build runs check_zed_phoenix.py with networking disabled. It exercises real installed auth stamping/body-carrier rejection, custom singleton loading, Router/Responses SDK dispatch, Responses SSE iteration, private SDK/protobuf serialization and a synthetic loopback OTLP receiver. Provider HTTPX transport is replaced by synthetic responses; no DB, provider or Phoenix is contacted.

The runtime unit suite is tests/test_zed_phoenix.py (requires installed LiteLLM and OTel). Run it in an already-local candidate image with --pull never, --network none, --read-only, a bounded /tmp tmpfs, service sources mounted read-only at /capture, tests at /tests, and PYTHONPATH=/capture:

python -m unittest discover -s /tests -p test_zed_phoenix.py -v
python -O /capture/check_zed_phoenix.py

These checks cover concurrent Zed/Companion fixtures, forged identity, request privacy controls, long history/tool catalogs, usage, output projections, retries, complete/incomplete/error streams, bounded state, quiet transport failures and fresh-process shutdown. Source contracts also test callback registration, disabled activation, model semantics, build-context isolation and fingerprint coverage. They do not prove DB-backed key authentication, OpenAI compatibility, Phoenix retention/UI fidelity or actual Zed integration.

Build-guard deadlines and safe diagnostics

Komodo Update 6ae7ed7636b469593dbccb1a failed during Compose Build, with Zed content guard FAILED: deadline; the existing proxy was not replaced. Native host diagnostics measured about 53 seconds registering FastAPI/Pydantic routes, with the first synthetic provider call reached only around 60 seconds. The old 60-second watchdog expired during dependency initialization, not on a failed isolation assertion. ARM/AMD64 tests on the workstation had not exposed that native-host cold-start cost.

The source fix retains every behavioral assertion and --network=none. The watchdog is now 120 seconds, with a 60-second exit-child timeout instead of 25 seconds. The child tests real typed-auth callback gating, private SDK export and interpreter-exit drain without reusing the full proxy request-setup fixture; parent tests still exercise the real auth adapters and SDK/Responses paths. The runtime capture callback, LiteLLM pin and capture policy are unchanged.

Failures include an allowlisted phase: imports, callbacks, native-api, exit-flush or verify. Known assertion labels may be shown; arbitrary third-party exception text, child output and tracebacks remain suppressed. For example:

Zed content guard FAILED: deadline [phase=native-api]
Zed content guard FAILED: fresh-process exit timeout [phase=exit-flush]

The main watchdog begins after Python startup; its 120 seconds is not a total build wall-clock limit. An isolated native-host candidate passed all 14 expected spans in 142.63 seconds including interpreter startup. This was a disposable, network-disabled test, not a successful Komodo deployment or live tracing test. Further host stalls can still fail the finite deadlines.

If a build still fails, record the exact Update ID, source revision, safe phase/ check label, elapsed time and host resource pressure. Do not enable detailed-debug payload logging, bypass/remove the guard, or keep increasing deadlines blindly. After the fix is merged, verify the build consumes that reviewed revision and refresh/check its existing deployment fingerprint before a separately approved, scoped retry. A successful local image or isolated guard does not complete the owner's deployment/acceptance checklist.

Owner to-do list (in order)

Work top to bottom; stop if a prerequisite is incomplete. An unchecked item means not verified, not permission to proceed. The owner makes privacy/spend decisions and performs credential/editor UI actions; an authorized operator or agent can do the host work. Instructions to request work below are not authorization already given, and this checklist does not execute anything.

Keep dates and non-secret outcomes with each item when it is completed. Never put keys, key hashes, decrypted env files or captured conversations in the checklist, chat, screenshots or PR. Do not check a box based only on a successful build.

1. [x] Merge the implementation.

Fulfill: PR #569 is merged. Its source, callback, offline tests and runbook are available from main; no second gateway or version upgrade is required.

Done when: GitHub shows the PR as merged. This does not prove that the running proxy contains the callback or that Zed uses LiteLLM.

1. [ ] Choose privacy, retention and spend limits.

Fulfill: Tell the operator the monthly Zed-key budget, budget reset period, desired rate limits, and maximum total spend for the synthetic acceptance run. Choose a trace-retention period; seven days is a proposed pilot, not a configured default. Confirm that full message/tool text may contain private code or accidental secrets and that OpenAI still receives inference content. Do not describe a gateway budget as a guaranteed OpenAI billing hard cap.

Decide explicitly whether to accept the existing unencrypted LAN HTTP collector hop for this pilot, or require encrypted transport first. The latter requires a separately reviewed change to the callback's fixed endpoint/transport; simply changing a Phoenix environment URL does not redirect this exporter.

Done when: The selected limits and transport/privacy decision are recorded without secrets. Do not enable capture while those decisions are outstanding.

1. [ ] Verify Phoenix access, local-only storage and retention.

Fulfill: Open https://phoenix.infra.realemail.app through Authentik. Ask the operator to verify the actual Phoenix version, native authentication/roles, collector authorization, free disk space, telemetry/export destinations, and effective backup policy. Review who can read the new zed project; a project name alone is not an access boundary. Keep the existing server credential in secret storage, not this document.

Have the operator confirm how the installed version implements the retention chosen in item 2, configure it only with approval, and verify readback. If it lacks suitable per-project retention, stop and agree on a supported scoped alternative; do not change shared-project retention or run ad hoc SQL/deletes. Confirm retained copies/backups also stay local. The source tier-2 policy says offsite: false, but inspect deployed jobs rather than treating that as proof. No offsite/cloud trace copy or backup deletion is authorized by this list.

Done when: Authorized access, an effective retention mechanism, local-only backup/export behavior and storage headroom are verified. Record safe results, not credentials or historical trace contents.

1. [ ] Approve a rollback-ready, capture-disabled proxy deployment.

Fulfill: Ask the operator to inspect the actual running Compose directory, image and private secret source; do not assume /opt/homelab is the active release or reset its dirty checkout. Preserve the matching pre-change image/config/secrets and take/verify a consistent recoverable LiteLLM DB backup using the deployment/recovery contract. Confirm the existing upstream OPENAI_API_KEY is the intended owner credential without printing it; if it must change, use SOPS under separate approval.

Approve a maintenance window and the exact reviewed source revision to build and recreate only the existing LiteLLM proxy, retaining its pinned version, DB container and other clients. Explicitly keep ZED_PHOENIX_CONTENT_ENABLED=false; leave the capture hash unset. Require the offline image guards and scoped input refresh/check before cutover. This is not permission for a broad deploy, DB recreation, migration experiment or version upgrade. Stop if unexpected startup migrations are proposed.

Done when: The operator verifies the deployed image/revision, health, unchanged global capture settings and existing-client configuration, plus availability of both gpt-6.1-sol and gpt-5.6-luna. Capture must still be off. Deploying the aliases first avoids trying to create a key for unavailable models.

1. [ ] Create and securely save the dedicated Zed virtual key.

Fulfill: After item 4, open https://litellm.infra.realemail.app/ui and use its virtual-key administration, or approve the operator to do this internally. Create one client key with alias zed, restricted to exactly gpt-6.1-sol and gpt-5.6-luna, with the budget/reset/rate limits from item 2. Do not use a master key, wildcard model grant or administrative role in Zed. Do not reuse or edit the Companion key. If a Zed key already exists, inspect and reconcile it deliberately instead of creating a duplicate.

Set these key-owned metadata fields, preserving unrelated fields and checking for conflicting team project overrides:

{
  "zed_phoenix_content": false,
  "phoenix_project_name": "zed",
  "phoenix_project_name_override": "zed"
}

Save the newly issued bearer credential directly in 1Password for recovery; do not paste it into chat, settings JSON, shell arguments or repository files. Have the operator privately read back the effective model grants, limits, metadata and proxy-resolved SHA-256 key hash. Do not replace the resolved hash with the key alias or with a hash of an already-hashed value.

Done when: The restricted key is securely recoverable and its effective policy is verified. Its content marker remains false and capture remains off.

1. [ ] Approve Zed-only capture activation and the second proxy recreation.

Fulfill: Authorize the operator to merge the following into the existing encrypted LiteLLM secret source, preserving all other values and the DB password: ZED_PHOENIX_CONTENT_ENABLED as the string "true", and ZED_PHOENIX_KEY_HASH as the resolved lowercase SHA-256 from item 5. Retain the existing PHOENIX_API_KEY. Edit through SOPS; never put a plaintext key or decrypted secret file into Git. Review/publish only ciphertext changes under the repository's normal delivery rules.

Authorize changing only the Zed key's zed_phoenix_content metadata to JSON boolean true, then rendering/refreshing/checking the private inputs and recreating only the proxy in the agreed window. The hash is process startup configuration, so changing the secret source or key metadata alone is not sufficient. Do not set global message capture or payload/debug stdout flags.

Done when: All activation gates in the isolation contract agree for the dedicated key, health is verified, existing callbacks/key metadata are unchanged, and the original image/config remain available for rollback. Zed itself should still use its original endpoint until item 8.

1. [ ] Approve and review synthetic gateway/Phoenix acceptance.

Fulfill: Approve the synthetic spend ceiling from item 2 and ask the operator to use only fixture messages/images and securely injected test keys. Require native /v1/responses, streaming, a model tool call and synthetic result follow-up, images, Priority/reasoning/cache parameters, and compaction where Zed uses it. Include a 60-turn history and 20-tool catalog with first/middle/last sentinels, an oversized field, a controlled error and an interrupted stream. Returned cache hits are not guaranteed.

In Phoenix, inspect project zed for zed.content.completed and exact response/call attribution, JSON messages, complete in-budget tool schemas, arguments/results, exposed usage/cost, timing and explicit omission markers. The project may first appear on ingestion; its presence alone is not a pass. Expect the existing metadata-only observation as well; do not call it a duplicate execution or a merged end-to-end agent trace. Review logs only for synthetic sentinels, without dumping other clients' request bodies.

Ask the operator to verify no new content export for a synthetic non-Zed request using an authorized test key, and that existing capture settings, client key metadata, export destinations and backup locality are unchanged. Do not replay real Companion/archive traffic or create another test key without approval. If no suitable test key exists, stop for that approval.

Done when: A bounded, secret-free acceptance report records all results and gaps, with no cross-key capture or unexpected destination. Stop on leakage or incompatible API behavior; do not move Zed onto an unverified gateway.

1. [ ] Set Zed's built-in OpenAI URL and gateway credential together.

Fulfill: Pause native-agent work and use Zed's command palette to open its user settings file. Preserve the original endpoint setting/absence and original OpenAI keychain credential for rollback. Ask the operator to check whether the running Zed process inherits OPENAI_API_KEY, reporting only presence/absence. Do not use env, process-environment dumps or commands that print its value. If set, remove the override from the actual launch/secret- injection source and restart Zed after saving work; unsetting it in an unrelated shell does not alter an already-running app.

Merge only this field into existing settings; do not replace the file:

{"language_models":{"openai":{"api_url":"https://litellm.infra.realemail.app/v1"}}}

Before sending any prompt, run agent: open settings, open the built-in OpenAI provider's credential controls and enter the Zed virtual key for the new URL. Use Zed's credentials UI/system keychain, not a custom Authorization header. Never enter the upstream OpenAI key as the gateway credential. Preserve the built-in Sol model, Fast/Priority, medium effort and all unrelated settings; do not add custom model redefinitions. Do not use existing chats for validation.

Done when: The gateway URL and its keychain credential are paired, no environment credential overrides them, and no model/feature settings were unintentionally changed. This covers built-in OpenAI-provider uses, not other providers, external agents or edit-prediction services automatically.

1. [ ] Run one new synthetic Zed-native conversation.

Fulfill: Open a new fixture-only workspace and a new native Zed Agent thread, selecting built-in OpenAI gpt-6.1-sol. Do not attach real source, transcripts, archive data or old conversation history. A suitable tool fixture prompt is: "Run printf 'zed-observability-fixture-42\\n' in the terminal, then report its output. Do not read files or call other services." Approve only that synthetic tool action. Also exercise a synthetic thread title/summary or commit-generation fixture to check the Luna route where applicable.

Open the matching new observations in local Phoenix and verify the endpoint, model, streaming output, tool call/arguments and result in subsequent context. Check native UI/JSON readability and keep only bounded non-secret outcome/ID evidence. A successful direct API request from item 7 is not a substitute. On 401, model/stream/tool failure, missing capture or wrong-project capture, stop and use rollback rather than testing with a real chat.

Done when: Actual Zed-to-LiteLLM-to-OpenAI-to-local-Phoenix behavior is verified and limitations are recorded. Local tool timing, subagent parentage and compaction lifecycle are still outside this callback's execution boundary.

1. [ ] Accept the pilot, then monitor retention and storage.

   Fulfill: Review the acceptance report and explicitly authorize normal content-bearing use only after items 2–9 pass. Have the operator record the deployed image/revision, verified versions, limits, retention/backup behavior, test outcomes, known gaps and rollback readiness in the runbook/journal; do not publish payloads, secrets or key hashes. Only then mark live acceptance complete. No historical chat import is authorized.

   Check the first day's actual Phoenix DB/volume growth and verify the chosen retention takes effect after its interval. Repeated conversation context can make storage grow much faster than chat-history size. Agree on a recurring capacity review and stop/disable capture if growth or retention differs from the approved policy. Never remove projects, backups or shared data as an unapproved cleanup shortcut.

   Done when: The pilot has recorded owner acceptance and scheduled follow-up; retention/storage checks have their own dated evidence, not just a plan.

Rollback

Restore the original OpenAI base URL/absence of override and select its original keychain credential, without overwriting unrelated settings. Revoke the new virtual key to stop its traffic, or remove its marker to stop optional capture. To disable capture process-wide, set ZED_PHOENIX_CONTENT_ENABLED=false in the private source, render/check and recreate the proxy under separate restart approval. Already queued spans may still drain. Existing traces/backups remain; no automatic deletion, project migration or retention change is performed.

Retain the pre-change proxy image/config/secrets. No version upgrade is intended; if startup nevertheless changes the DB, stop and use the matched recovery procedure rather than running an old image against an unverified schema.

Agent-execution boundary

Gateway spans do not directly time local terminal/file/MCP execution or establish Zed thread lifecycle, subagent parentage or compaction events. Prompt-cache keys may correlate some requests but are not a complete session trace. Previously discarded context and historical chats are not recovered. No static traceparent is added. Zed-side instrumentation or separately approved transcript enrichment would be a future task, not part of this callback.