Nocturne recovery and publication source milestone

Recovery deltas no longer need to wait for baseline snapshot completion when the full epoch prefix is available. Snapshot publication and event queue accounting now use bounded work and explicit recovery semantics. This is source-only work: no helper, gallery, scraper, worker count, mount, or sync mode was changed.

Baseline and access

• Isolated a task worktree from refreshed main at c157f94d; unrelated work in the original checkout is untouched. PLAN.md is unchanged.
• Privately inspected the copied memory-policy, controller-status and spot artifacts. They confirm the one-hour initial-recovery deadline, no observation start, and the previously passed isolated spot checks. They are historical evidence, not fresh runtime samples. Private artifacts remain ignored.
• A bounded NAS SSH probe passed with strict host-key verification and agent forwarding disabled. An exact-process probe found no helper/controller script process. This does not prove privileged container state: noninteractive Docker inspection failed with sudo: a password is required.
• Owner-provided privileged inspection subsequently confirmed the NAS helper is stopped (Exited (137)), OOMKilled=false, restart count zero, CPU affinity 7, RAM and combined RAM/swap limits both 1,073,741,824 bytes, read-only root, writable state and read-only originals. Exit 137 alone does not establish OOM. Its image ID is sha256:c995a16c6af8c7907033e5a683b942c3cdc5971357ced9b3b65ec56e48a71692.
• Owner-provided PC output confirmed nocturne-gallery:5953ea3b-pc running and healthy. Worker count, sync mode, scanner progress, scraper counters and external controller state were not freshly verified. Last recorded production remains legacy with ten workers. No new remote tree walk ran.

Implemented and tested

• Register Watchman and obtain an explicit fresh clock; never start with an unfiltered cached-file query. Record phase/progress timing while preserving the original recovery start.
• Drain changes between traversal entries, staging batches and finalization transactions. One blocking filesystem operation can still delay polling.
• Bootstrap only at sequence zero with a fully retained epoch prefix. Apply contiguous batches and catalog checkpoints atomically, at most 400 records. A provisional cursor does not establish a healthy baseline; gaps reconcile.
• Protect newer delta paths, including move origins, during snapshot paging. Preserve annotations, preview state, and shadow's zero-job behavior.
• Bound overlay, comparison/merge, immutable snapshot materialization and old generation reclamation. Empty/incomplete inventories do not authorize absence. Retire a generation before deleting any published rows; readers then get a gap.
• Maintain transactional queue event/byte counters, indexed oldest-event lookup, additive migration/backfill and rollback safety. Queue pressure invalidates an active snapshot before discarding its required history. Preserve one writer and the existing SQLite journal mode.

flowchart TD
    Register[Register source and establish clock] --> Delta[Persist ordered changes]
    Delta --> Provisional[Apply contiguous provisional batches]
    Register --> Inventory[Independent staged inventory]
    Inventory --> Batches[Bounded finalization with delta draining]
    Batches --> Publish[Atomic immutable generation]
    Publish --> Baseline[Validate consumer baseline]
    Provisional --> Catalog[Usable catalog and annotations]
    Baseline --> Catalog

Validation and measured limits

The Linux Docker dependency/test image ran current source through read-only mounts: 234 backend tests passed, including acceptance/activation helper regressions; one existing Starlette/httpx deprecation warning remains. The initial parent harness used a non-executable temporary filesystem, which prevented a fixture shell command from running. Rerunning with executable temporary fixture storage passed. This is not a newly built candidate-image test result.

The release contract suite passed all 13 tests. A first invocation also named a nonexistent deployment-test module; the corrected release-only command passed.

The reproducible app/tools/benchmark_discovery.py fixture used 100,000 synthetic metadata records, no originals and no Watchman. It measured 758 finalization callback/batch iterations, zero new events for unchanged finalization and zero media-row updates during unchanged catalog application. Sampled annotations and preview flags remained intact; integrity checking passed. The recorded JSON identifies uncommitted source at measurement time and local x86 emulation. Health probes ran between committed batches; they do not establish maximum concurrent HTTP latency or lock-hold time. This is not a comparable full reconciliation, source-operation benchmark, or NAS/scraper performance result.

Blocked and deferred

• The initial discovery-image build failed because the rolling Homebrew API returned Watchman 2026.10.26.00, not the pinned 2026.10.19.00. The build now disables API installs and shallow-fetches the immutable Homebrew/core commit containing the existing version/bottle, verifying checkout identity before metadata checks and installation. The base, Watchman version and bottle digest guards are unchanged. The new x86 image built successfully; the image/source manifest verifies all three helper backend files against that exact image. This pins formula provenance, not every external apt/pip build input.
• One small actual-Watchman fixture passed against that image with one CPU and one-GiB combined memory/swap on local cgroup v2. Quiet-folder discovery measured about 5.39 seconds; settling, identity-preserving rename, deletion, restart recovery, read-only enforcement and excluded-warning handling passed. See the smoke evidence. No production media/state was mounted. This does not establish NAS cgroup v1 behavior, full-library performance or the bounded live acceptance window.
• Root-level build contracts cover immutable source ordering, synthetic pin bumps and unsafe install mutations; they run in repository CI, not the packaged app tests. The corrected contract plus release suites passed sixteen tests.
• The configured release workflow permits image builds only on a manual main dispatch. PR contract success cannot substitute for that workflow. No release or deployment was dispatched from this task branch.
• Cached metadata reuse, fewer ancestor operations and stable identity across helper/container recreation remain deferred. An attempted descriptor-based path increased metadata operations and was removed rather than presented as an optimization. Existing identity and filesystem behavior is retained.
• Startup/reset snapshot cleanup still has bulk operations; finalization's hot path is bounded. Full NAS baseline/parity, every scraper write method, the provisional twofold target, five-percent scraper gate and new live spot checks remain unverified.
• Unattended production transport, restore-tested backup/migration, exact image hashes and rollout/rollback verification remain activation prerequisites.

The owner supplied the read-only NAS inspection and subsequently merged PR #556 as 8ff77d19. They approved one quick shadow attempt, not production activation. A separate foreground helper-only diagnostic is prepared with a hard 600-second runtime, independent stop guardian, exact image/archive/source checks and fresh state. No source fixtures or catalog consumer are involved, so this diagnostic cannot establish discovery-to-catalog or scraper acceptance. At preparation time no new helper was started: this Mac still lacks owner-authenticated NAS sudo. The verified image archive and source were staged privately for the owner start. The owner's first start refused during image verification, before a trial helper was created: modern Docker's OCI index ID is not an image lookup ID on the older NAS engine. The corrected loader verifies the hashed archive chain through the Linux x86 manifest and config, checks the loaded RootFS/platform, and uses the verified configuration ID on classic Docker. Read-only parsing on the actual NAS confirmed that chain; no image re-transfer or trial start was needed. Fifteen launcher tests cover modern/classic identity and mismatched metadata rejection. The transfer's first bounded command timed out; its remaining bytes were resumed and the complete remote archive hash was verified. No old archive was replaced.

Completed bounded helper diagnostic

On 2026-10-31 UTC, the owner supplied the final JSON from the single corrected 600-second run. It completed with no reported error and confirmed the new trial container stopped. The helper retained its fresh state; no production catalog consumer, fixture or gallery change was involved. See the aggregate evidence.

The final sample at 598.721 seconds reported 367,200 staged records, 488 retained events, floor zero and state recovering/staging. Watchman registration took about 58.52 seconds and checkpointing 0.31 seconds. Cumulative staging time was 131.73 seconds; the latest query/metadata/persistence measurements were approximately 0.122/0.011/0.131 seconds, not whole-run percentiles. This demonstrates event persistence during inventory work, not event-to-catalog/search latency.

Final resource values were zero memory-limit hits, no OOM/restart and no pressure streak; usage was 404.81 MiB, RSS 254.85 MiB and swap 1.65 MiB. The last major-fault delta was zero. Incidental swap is allowed within the approved cgroup-v1 combined cap. The report counted 118 samples, but only the terminal summary/last sample was supplied; full time-series analysis was not performed. last_state.running=true is the sample before shutdown, not a contradiction of later stop_confirmed=true.

No healthy inventory completed (last_reconciliation=0), and finalization, full-catalog parity, a 600-second healthy observation, scraper impact, comparable twofold speedup and production transport remain unverified. Do not compare the staged-row checkpoint directly with the earlier legacy scan as a controlled speedup. The bounded attempt is finished; no extension or repeat is scheduled. Merge and production activation require separate approval. Never deploy this source with the previous helper image.