Skip to content

Nocturne metadata and batching efficiency

Source-side metadata access, identity stability and SQLite batching have advanced without another NAS run or production activation. New helper/gallery images built locally and passed bounded fixture validation. Full NAS baseline/parity, searchable delivery, scraper impact and production transport remain open gates.

Repository repair and isolation

The malformed refs/remotes/origin/main 2 pointed to a valid older commit but blocked fetch. Its original bytes were quarantined outside the ref namespace in the shared Git directory; the canonical ref and user work were not overwritten. Fetch and git fsck --connectivity-only --no-dangling then passed. The duplicate's cause is unproven. Unrelated duplicate working files also appeared during this work and were preserved; final images were built from the exact staged Git tree, excluding those untracked copies. Interim worktree test counts were not used as canonical results. The new branch starts from merged main 7917800c and preserves all unrelated work and previously retained trial artifacts. PLAN.md is unchanged.

Implemented source behavior

  • Staging checks up to 400 distinct paths in one lookup instead of 400. Duplicate input paths increment the staged count once, with last observation winning.
  • Finalization batches current metadata and newer-event provenance. Both event path indexes are searched; distinct result size stays at most 400 names, with at most 802 bindings. No SQLite journaling or transaction-bound change.
  • Targeted changes reuse bounded pinned directory descriptors instead of repeatedly checking every ancestor per file. Metadata-only descriptors require search access, not extra directory-read permission. Wide folder bursts flush smaller batches at the real descriptor budget without advancing the source clock until the complete query is durable; old move origins use separate short-lived contexts. Whole-folder moves preserve IDs and annotations. Metadata and trustworthy identity come from the same O_PATH leaf descriptor, without reading original bytes.
  • Inventory streams directory entries and revalidates directory mappings around publication. In-scope symlinks conservatively make an inventory partial, including names resembling media files; hidden/support exclusions still apply. Directory safety checks add operations and are not claimed as free speedups.
  • A growing file no longer invalidates its entire batch because its version changes after observation. The ordered journal covers subsequent leaf changes; directory/root replacements still fail closed. This is not an atomic filesystem snapshot or a guarantee against a single blocking filesystem call.
  • Staging progress writes are throttled to five-second intervals, retaining final totals. The thirty-minute independent-inventory interval is unchanged.
  • An invalid Watchman clock returns an empty fresh-instance response rather than materializing the entire cached tree; explicit epoch recovery still follows.
  • Watchman requests cached size and nanosecond mtime, but those fields remain advisory. A trustworthy source observation is still required; eliminating that observation remains deferred.
flowchart TD
    Source[Read-only source] --> Pinned[Bounded pinned directories]
    Pinned --> Leaf[Same-descriptor metadata and export handle]
    Scope[Persisted UUID and stable filesystem namespace] --> Leaf
    Leaf --> Delta[Ordered durable changes]
    Pinned --> Staging[Bounded inventory staging]
    Staging --> Merge[Indexed batch comparison]
    Delta --> Merge
    Merge --> Snapshot[Complete immutable generation]

Identity and migration boundary

Collector identities use a durable UUID source scope plus a supported filesystem FSID and complete generation-bearing export handle, not a namespace-local mount ID. Root device validation is cached only within the pinned context; device IDs are not persisted as identity. Supported Linux Btrfs subvolumes use distinct anonymous devices, including on kernel 4.4, so mismatched child devices fail closed. Per-file statfs is avoided with that trusted context. Unsupported filesystems/handles, hard links and namespace mismatches yield no identity.

Moving from legacy identity to v2 may emit one-time identity/settling changes. There is no speculative legacy-to-v2 rename bridge without prior namespace proof; a rename crossing that transition may remain separate entries with the old annotations retained. Unchanged size/mtime does not reset media derivatives. Existing production polling mode and workers were not changed.

Review reproduced a same-path replacement with different contents but identical size/mtime: the old previews and hash stayed valid, and a stale worker could publish. A change to tracked metadata or trusted v2 identity (including feed identity loss) now rotates a tagged token in the existing media.generation field, clears derived readiness/jobs/hashes/retries, and preserves annotations. This also avoids cache-key collisions when a detected mtime change falls within one microsecond. Metadata-only fingerprints remain a heuristic, not content hashes. Cache names include only that reserved token; legacy generation values and legacy-to-v2 migration keep their old names. Worker metadata, readiness, hashes, errors, retries and completion are fenced by captured size/mtime/generation and claim. Browser derivative URLs carry the same source-version stamp; stale or unversioned requests revalidate instead of retaining day-long cached pixels. Two Node unit tests and HTTP regressions cover this non-visual change; no CWV or browsing-speed claim is made. A deleted source also retires its exact stale claim, preventing a stranded running hash job from blocking the hash lane. No database column or journaling change was needed.

Legacy state cannot prove an unchanged same-metadata file across migration, so activation must drain legacy workers. Feed rollback should retain the new generation-aware gallery image and switch scheduling to legacy; an older image that ignores source-generation cache keys needs separate compatibility review.

The actual two-container Collector proof changed the mount namespace and mount ID while retaining the scope/identity. A completed one-file reconciliation emitted zero new feed events, zero media updates and zero preview jobs; the catalog digest was unchanged. This passed on a disposable local ext volume, not the NAS Btrfs filesystem. The final fast-path proof is recorded in Collector identity evidence.

Measured validation

  • First committed store-only staged snapshot: 241 backend tests passed.
  • Final exact locally built gallery/test image: 295 passed, 8 skipped, one existing Starlette/httpx deprecation warning. Two skips need Docker CLI; six need supported filesystem/export handles in the temporary source filesystem.
  • Separate exact-image identity suite on a disposable ext volume: 12 passed.
  • Actual two-container Collector/catalog proofs: 2 passed, with a host pytest/anyio deprecation warning. No Docker socket was exposed to the packaged test container.
  • Root build, launcher and release contracts: 35 passed.
  • Small actual-Watchman fixture on the exact new helper: about 4.36 seconds quiet-folder discovery; settling, identity-preserving rename, deletion, restart recovery, read-only enforcement and excluded-warning handling passed under one CPU/one-GiB combined limits on local cgroup v2. Not a NAS resource/latency result.
  • One 100,000-record SQLite fixture: complete immutable snapshot, zero new events from unchanged finalization, zero media updates, preserved sampled annotations and previews, integrity OK. It ran on the final generation-aware candidate under one CPU/one GiB, with no media traversal. Health probes were between batches, not concurrent requests under lock. No controlled baseline comparison or end-to-end twofold speedup is established.

SQL operation evidence records 400-to-one staging lookups and 1,200-to-two merge metadata reads without identities (three with identities). Source operation evidence counts Python-visible calls: 249 versus 480 for 120 targeted files with a mocked supported v2 API. It excludes identity FFI and internal directory calls, and does not establish total syscall savings or wall-clock speed. Inventory adds directory safety work; no lower-total-operation claim is made.

Exact images and release test repair

The new image/source manifest binds all four helper backend modules to the locally built helper, gallery and test images. Integration was uncommitted when built; checksums identify the exact bytes. This is not publication through the main/manual release workflow.

Testing the actual packaged image exposed an existing activation-fixture layout bug: the service helper was outside the app build context, and shallow /tests paths could not resolve it. The test build now uses service-root context and includes the real helper, peer tools and an inert executable Docker logger for the no-exec temporary filesystem. Test-only assets remain outside the production image. The original read-only/network/capability restrictions and activation gates are retained. An initial packaging run failed five fixtures; the corrected exact image passed the suite. Subsequent generation-fencing regressions passed in the final rebuilt candidate; the old pre-fencing SQLite measurements were retained privately rather than presented as current results. The helper-image contract also verifies local imports are copied, including the new identity module.

The bounded launcher accepts either a direct OCI image manifest or an OCI index, verifies the hash chain/RootFS in both cases, and checks every declared supported source hash, including the optional new module. No new image/archive was staged on the NAS during this work.

Remaining boundaries

No NAS restart, source fixture, scraper write, production release dispatch, credential generation or activation occurred. The previous short diagnostic is still the only new live checkpoint. A new NAS source/image bundle and any further short run require separate review/authorization. Production unattended encrypted transport and credentials, actual scraper-method coverage, full baseline/parity, credible scraper impact, backup/restore/migration and user-facing activation tests remain outstanding. The explicit 600-second policy remains; no day-long wait.