Nocturne discovery performance implementation plan

Resume point and owner decisions

Implement this plan from current origin/main on a new task branch. This is a follow-up to PR #547, merged as 8fbc43fe on 2026-10-30. That merge delivered prototype and validation support; it did not activate the production feed. Do not edit the repository's PLAN.md.

The owner wants substantial discovery improvement, with these priorities:

1. New media searchable and browsable.
2. Faster full-library reconciliation.
3. Thumbnails and previews.
4. Browsing responsiveness.

Healthy discovery limits remain five minutes for active folders and ten minutes for quiet folders. Aim for delivery within thirty seconds during ordinary event processing. Recovery must be visible and can take longer; it must not silently block every new addition behind a whole-library inventory.

The owner explicitly replaced the mandatory twenty-four-hour observation with bounded spot validation. Use the explicit 600-second policy already added in PR #547. Do not introduce another day-long wait. Keep other correctness checks and report unmeasured performance or scraper-impact claims honestly. Merge and production activation are separate decisions.

Source implementation checkpoint

The recovery/publication milestone implements Phase 1 scheduling and provisional contiguous delivery, bounded snapshot publication, transactional queue accounting, and focused recovery tests. The 100,000-row SQLite fixture is not a NAS reconciliation benchmark. Phase 2 metadata reuse and stable identity namespace remain deferred; filesystem rules are unchanged. The pinned formula-source rebuild now succeeds and one small actual-Watchman fixture passed locally; new live spot checks remain pending. Owner-supplied inspection confirms the stopped NAS helper's resource/read-only settings and the expected healthy PC image; worker count and sync mode have not been freshly inspected. No deployment, activation, completed NAS baseline, production transport, or scraper-performance acceptance is claimed.

Verified starting state

These are recorded checkpoints, not a substitute for refreshing live state:

• The PC deployment uses nocturne-gallery:5953ea3b-pc, ten workers and GALLERY_SYNC_MODE=legacy. The older saltierpoop gallery remains stopped.
• Originals are mounted read-only; catalog/cache remain on PC storage. The existing NFS entrypoint is required on Docker Desktop. Do not replace it with an unverified host-mount propagation arrangement.
• A live legacy scan reached 366,800 unchanged items in about twenty minutes, with about twenty-eight seconds in catalog application. This was a progress checkpoint, not a completed comparable reconciliation benchmark.
• NAS one-CPU/512-MiB discovery failed its resource trial. The owner approved one CPU and one GiB for both RAM and combined RAM/swap on the subsequent trial.
• Synology kernel 4.4/cgroup v1 lacks the CPU CFS control used by cpus; CPU affinity [7] was verified. The old kernel permits incidental global paging within the combined cap. Do not promise absolute zero swap.
• The one-GiB attempt reached its one-hour initial-recovery deadline without a healthy completed inventory. The controller and helper stopped with state retained. All 110 resource samples had zero memory-limit hits. This does not demonstrate another RAM-capacity failure.
• Live NAS spot fixtures passed additions (about sixteen seconds), growth, settling, rename preserving ID and annotations, deletion and duplicate replay into a separate partial catalog. Production received zero fixture rows and the test catalog scheduled zero derivatives. Cleanup was verified.
• Full-catalog parity, every scraper access method, comparative reconciliation speed, scraper impact and production transport remain unverified.

Read the deployment record, service page, and aggregate spot evidence.

Scope and reuse

Keep SQLite and its current journaling mode, one production catalog writer, the existing hosting arrangement and ten media workers. Scrapers and their configuration stay unchanged. No whole-library hashing, decoding during discovery, replacement database, distributed workers or public discovery route.

Reuse Watchman, the durable NAS feed, additive catalog sync schema, separate shadow CLI, existing settling logic and the release pipeline. Extend these interfaces instead of introducing a second independent scanner or production patch. The new work is startup/recovery scheduling, metadata reuse, bounded snapshot publication, cheap feed health and validation instrumentation.

flowchart LR
    W[NAS Watchman] --> Q[Durable ordered changes]
    Q --> C[Consumer checkpoint and catalog transaction]
    C --> L[Searchable live catalog]
    I[NAS ground-truth traversal] --> S[Staged inventory]
    Q --> S
    S --> P[Healthy atomic snapshot publication]
    P --> C
    L --> D[Settled derivative work]

Phase 0: refresh and isolate

1. Read AGENTS.md, project conventions, service README and DEPLOY.md. Fetch main, inspect branches/worktrees and preserve unrelated work. Use a dedicated topic branch; commit/push reviewable slices and maintain one focused PR.
2. Refresh the deployed image, worker count, sync mode, scanner progress, helper state and existing controller processes. Do not trust old PID files.
3. Inspect private operator artifacts under .scratch/nocturne/, especially hash-trial/memory-policy-ready.json, shadow-runtime-memory-trial-2026-10-30/controller-status.json and spot-validation-2026-10-30/report.json. Keep raw paths/logs/tokens private.
4. Verify NAS access and existing resource controls before another start. Privileged NAS Docker requires owner-authenticated sudo; do not change SSH policy, groups or sudoers to avoid that boundary.
5. Preserve the existing source/image archives, state and backups. A new backend requires a newly tested image and matching source/checksum manifest. Do not combine new source with the old image or create another production override.

Deliver a short baseline record using existing scanner/status/scraper counters. Avoid launching another remote tree walk just to collect baseline data.

Phase 1: separate live changes from baseline recovery

Primary files: app/backend/discovery.py, sync_client.py, catalog_sync.py, discovery_store.py, shadow_sync.py and their tests under services/nocturne-gallery/.

1. Add phase timings/progress for watcher registration/query, source metadata, staging, catalog/store persistence, finalization and lock wait. Preserve the original start time. Expose progress without scanning the queue to compute it.
2. On a new recovery epoch, establish a Watchman checkpoint after registering the source. Avoid the initial unfiltered query followed by describing every cached path before ground-truth traversal starts. The independent inventory supplies the baseline; the ordered journal covers concurrent changes.
3. Interleave event draining with bounded inventory work. Persist events before advancing the Watchman clock. Keep discovery ahead of inventory and previews.
4. Let the consumer apply valid contiguous changes to an existing catalog while baseline recovery is incomplete. Commit each batch and its checkpoint together, at most 400 items per transaction. Keep status explicitly recovering.
5. Define bootstrap and gap semantics before coding. Never seed a checkpoint at current head or retention floor just to bypass missing history. An epoch gap or missing retained prefix still needs reconciliation. Retain a clear distinction between provisional recovery work and a validated baseline.
6. Existing entries/annotations remain usable. No unseen-entry deletion is justified by an incomplete, unreadable, empty or interrupted inventory.

Acceptance: with a deliberately slow baseline, a new quiet-folder addition reaches the isolated catalog within the normal discovery limit. Recovery remains visible; no skipped sequence, lost annotation or speculative deletion occurs.

Phase 2: reduce filesystem work

Primary files: discovery.py and sync_protocol.py.

• Request Watchman's cached size and nanosecond modification time for routine changes. Verify fields against the pinned Watchman version. Do not hash/decode.
• Avoid repeated ancestor lstat calls for every file. Reuse validated directory traversal state or safe directory-relative operations. Preserve containment, symlink and collection/extension/hidden/support exclusions under concurrent directory changes; do not trade these checks away for speed.
• Use targeted source reads for uncertain metadata, settling and trustworthy identity. Settling still requires two unchanged observations at least thirty seconds apart and remains a heuristic.
• Audit identity stability across container/helper restarts. The current file_identity includes a mount ID: test whether that namespace-local value causes unchanged files to appear changed after recreation. Use a trustworthy filesystem namespace plus export handle/generation where supported; never infer a move from matching size/time or inode alone.
• Ground truth performs one NAS-local traversal. Reuse directory-entry stat results and avoid a duplicate whole-library Watchman/metadata pass. Preserve the thirty-minute safety interval initially.

Acceptance: count metadata operations in fixtures, demonstrate no full remote walk in steady state, and retain move/symlink/permission correctness. Verify unchanged metadata across restart does not rewrite the whole catalog or rebuild previews.

Phase 3: bound publication and health costs

Primary file: discovery_store.py.

• Store.finish currently processes the full inventory under one store lock and transaction. Stage/compare/merge in bounded batches and keep HTTP health/change delivery responsive. Publish only a complete consistent snapshot generation.
• Preserve the traversal watermark and replay newer events correctly. Source changes during finalization must win over older staged observations. Never expose partially published membership as grounds for deletion.
• Compare metadata in batches and skip unchanged rows. Retain immutable snapshot paging and replay/crash safety. Do not make the feed checkpoint visible before the corresponding records are durable.
• Replace per-health whole-event-table COUNT/MIN/SUM work with transactional count/byte accounting and an indexed oldest-event lookup. Maintain these across insertion, pruning, epoch recovery and rollback; backfill existing state safely.
• Preserve one writer, additive migrations and the current SQLite journal mode.

Acceptance: large unchanged inventories produce no catalog rewrites; health and new-change delivery remain responsive during finalization. Crash/restart tests cover every publication boundary and accounting rollback.

Phase 4: bounded validation

Use focused tests per slice, then one representative large fixture and one live spot session. Do not repeat large trials without a new failure or changed code.

Check	Budget and required evidence
New and quiet-folder additions	Usually seconds; fail at five/ten-minute limits
Growth and settling	Observe changes and two stable samples at least thirty seconds apart
Rename and deletion	Verify IDs, notes, tags/albums and derivatives; test ambiguous identity separately
Duplicate/crash replay	Checkpoint and records remain atomic; no duplicate jobs or lost annotations
Gap/overflow/restart/outage	Explicit recovery; no checkpoint skipping or unseen-entry deletion
Partial/empty/unreadable source	Keep catalog usable; suppress unsupported deletions
Large unchanged inventory	Count source ops and DB writes; verify preview reuse and annotation digest
Source changes during derivatives	Stale jobs cannot publish readiness for a newer version
Live session	Target one ten-minute window; stop/profile a specific stall instead of adding a day-long wait

Use isolated fixtures outside scraper working directories. Prevent the legacy scanner from ingesting them, arm independent cleanup, and verify zero production fixture rows and zero shadow jobs afterward. Test writes through actual scraper access methods; a local fixture alone does not prove SMB/NFS behavior.

Use the source's configured test environment. Relevant checks include the backend suite, actual Watchman smoke tool and acceptance/activation tests. Run the configured Docker image test/release workflow before deploying a changed gallery image; a PR contract check is not an image-build result.

The explicit bounded checker is:

python app/tools/acceptance.py /private/path/trial-report.json --minimum-observation-seconds 600

The activation helper propagates that policy through DISCOVERY_MINIMUM_OBSERVATION_SECONDS=600; this is not permission to execute it. Its other gates still apply. Do not manufacture measurements to make it pass.

Report traversal, metadata, DB comparison/write, finalization, lock wait, total reconciliation and controlled discovery latency separately. The two-times reconciliation improvement remains a provisional engineering target under comparable load. Scraper regression must stay within five percent if measured credibly; otherwise label it unverified and present it at the activation review.

Phase 5: connection, release and activation

The merged fixed-endpoint SSH transport is shadow-only. Production still needs an unattended authenticated encrypted private connection. Verify that path from the actual gallery container, with reconnect/replay, before changing sync mode. Do not assume SSH forwarding works: it was prohibited on the tested alias. Do not put the owner's interactive 1Password signing setup into a container. Prepare any new connection/credential configuration for separate owner review; never silently weaken SSH restrictions or expose a public route.

Build and test the exact source/image, keep tokens private and stage a concrete rollout plus rollback. Keep the NAS trial within the approved one-CPU/one-GiB combined budget. Retain OOM/restart and sustained-pressure guards; additional resources need separate acceptance. Do not repeat the failed zero-byte swap rule.

Before activation, present passed spot checks, failed/unknown gates, the exact image/source hashes, private connection verification and restoration evidence. If a full NAS baseline remains incomplete, explicitly review any proposed rollout in recovering mode; never call it healthy synchronization or enable unsupported inventory-based deletion. Merge approval does not authorize activation.

With activation approval: make and restore-test a consistent catalog backup, validate additive migrations against a copy, stop the legacy scanner from concurrently applying results, and enable the new consumer with one writer. Verify search/browse additions, annotations, preview reuse, POST /api/scan and additive GET /api/status behavior at the user-facing layer.

Rollback disables the consumer and restores legacy scheduling. Preserve newly created annotations and unavailable entries; avoid restoring an old catalog over new user data as a routine rollback. Keep all recovery artifacts.

Suggested commit boundaries

1. Instrument phases and prove slow-baseline/fast-delta failure in a regression.
2. Implement fresh-clock startup and safe consumer delivery during recovery.
3. Reuse cached metadata and fix any proven identity-namespace instability.
4. Bound snapshot finalization with crash/concurrent-event tests.
5. Add transactional queue accounting and validate migration/replay.
6. Record large/live spot evidence, prepare the production connection and rollout.

Each slice includes relevant tests/docs, a coherent commit and push. Keep the PR scope and current-head checks accurate. Stop for a concrete owner decision only at an actual access, merge, resource or deployment boundary.

Prompt for the next session

Resume the Nocturne discovery performance plan in docs/runbooks/nocturne-discovery-performance-plan.md. Refresh current main and live state, then implement Phase 1 first. Preserve scrapers, read-only originals, annotations and the single writer. Use bounded spot validation, not a mandatory twenty-four-hour wait. Commit/push reviewable slices and maintain a PR; prepare activation for separate approval.