Skip to content

2026-11-02 — Phoenix 21.0.0 upgrade

Summary

Phoenix was upgraded from 20.26.1 to 21.0.0 on prox LXC 124 after a full backup/restore and isolated migration rehearsal. Production authentication, HTTP/gRPC ingestion, project routing, and historical trace checks passed; LiteLLM was not changed. After PR #576 merged, the host checkout was reconciled to 9111259f and Periphery plus both Ansible timers were restored without restarting Phoenix or PostgreSQL.

Times in this entry use UTC; the host-local operation date was 2026-11-01.

Completed

Gate Evidence
Backup 8,806,796,316-byte custom-format dump completed in 1,732 seconds; copied to infra-services with matching SHA-256
Restore Full restore on infra-services exited 0 against the production PostgreSQL 16.15 image; separate volume, internal-only network, no published ports
Rehearsal v21 migration reached 7c2bc8452d60; eight historical projects readable; canaries survived collector restart
Fresh recovery point Application and database cleanly stopped for CT 124 snapshot phoenix-pre-v21-20261101
Production Registry digest and package 21.0.0 verified; DB image and existing encryption secret unchanged
Authentication and ingestion Existing exporter key works; anonymous HTTP/gRPC ingestion rejected; both canaries persisted; HTTP header project routing passed
Historical data Existing project/user/API-key/dataset/experiment identities preserved; exact sampled historical trace IDs readable
UI surface Login page served and public HTTPS returned its authentication redirect; human browser login was not tested
Source reconciliation PR #576 merged; /opt/homelab clean at 9111259f, resolved v21 config unchanged; Periphery and both Ansible timers enabled/active at 07:21 UTC

No provider/model requests were made. Synthetic spans remain in project phoenix-upgrade-smoke. No production volumes, historical projects, or backup artifacts were deleted. Rehearsal containers are stopped; their volumes and verification reports are retained.

flowchart TD
    A[Completed database backup] --> B[Checksum verified off-host copy]
    B --> C[Full isolated restore]
    C --> D[Rehearsal migration and protocol checks]
    D --> E[Stop app and database]
    E --> F[Consistent CT snapshot]
    F --> G[Deploy pinned Phoenix 21]
    G --> H[Production protocol and historical checks passed]
    H --> I[PR 576 merged]
    I --> J[Host reconciled and automation restored]

Recovery and remaining work

The Phoenix service README records backup paths, checksums, recovery evidence, and the completed source reconciliation. Production evidence is under root-only /opt/phoenix-upgrade-2026-11-01-retry/live/; rehearsal evidence is under /home/someone/phoenix-v21-rehearsal-20261101/ on infra-services. The first 600-second backup attempt remains incomplete and must not be used for recovery.

The source-reconciliation follow-up is complete: the host's sole agent-owned Compose modification matched the merged v21 file byte-for-byte, the checkout fast-forwarded from c157f94d to 9111259f, and resolved Compose/container identities were unchanged. AUTOMATION_RESTORED.json records the three restored units; the retained hold note is marked RELEASED. This is not a claim that a subsequent scheduled Ansible apply/check has completed.

  1. Confirm a human browser login. API-key authentication and the login page passed, but this does not establish interactive sign-in success.
  2. An unfiltered old-project span listing exceeded a 15-second client timeout with PostgreSQL waiting on disk reads. Exact trace-ID reads passed; broad listing performance remains uncharacterized and was not changed by this work.
  3. Keep an eye on capacity: the guest had about 12 GB free. Proxmox thin-pool data/metadata usage was 87.71% / 3.07%, and snapshot creation warned that auto-extension protection is not configured. No capacity settings or backups were changed to address that separate concern.

An image-only downgrade against the migrated DB is unsafe. The consistent CT snapshot and matching pre-upgrade configuration are retained, but a full CT rollback was not executed; restoring it would discard writes since its recovery point. The earlier LiteLLM production acceptance is unchanged; this Phoenix-only rollout did not repeat its model tests or establish broader LiteLLM workflow acceptance.