feat(semantic-drift): streaming drift detection for agent vector memory (ADR-273)

[ruvnet]

Summary

Adds crates/ruvector-semantic-drift — three complementary streaming drift detectors that let MCP tools and index maintenance loops detect when agent vector memory distributions have shifted, without requiring offline batch analysis or O(n²) computation.

• WindowCentroidDetector: O(d) update, ~10M updates/sec, 1 KB memory, detects 8σ shift at lag 500
• ProjectionDriftDetector: k=64 Rademacher random projections, 251k updates/sec, 32 KB, lag 500; catches subspace shifts that centroid averaging dilutes
• SentinelQueryDetector: mean-all-sq-distance metric, 155 KB, lag 175 (fastest); stable metric replaces unstable Jaccard top-k overlap

All three share a DriftDetector trait with a poll_event() -> Option<DriftEvent> method (O(1)) that returns a DriftAction (Observe / Compact / Rebuild) for direct consumption by index maintenance schedulers.

Changes

• Cargo.toml — adds crates/ruvector-semantic-drift to workspace members
• crates/ruvector-semantic-drift/ — new crate (5 source files, 1 benchmark binary)
  • src/lib.rs — DriftDetector trait, DriftEvent, DriftAction, shared dataset module
  • src/centroid.rs — WindowCentroidDetector
  • src/projection.rs — ProjectionDriftDetector (Rademacher ±1/√d matrix)
  • src/sentinel.rs — SentinelQueryDetector (mean-all-sq-dist, circular snapshot buffer)
  • src/bin/benchmark.rs — acceptance benchmark: 5k baseline + 5k drifted 128-d vectors
• docs/adr/ADR-273-semantic-drift-ann.md — architecture decision record
• docs/research/nightly/2026-08-01-semantic-drift-ann/README.md — full research writeup with SOTA survey and design rationale
• docs/research/nightly/2026-08-01-semantic-drift-ann/gist.md — public-facing article

Benchmark Results (x86_64 linux, rustc 1.94.1, --release)

Variant              | Mean update | Updates/sec | Detect lag | FP | Memory
---------------------|-------------|-------------|------------|----|---------
WindowCentroid       | 100 ns      | 9,975,580   | 500        | 0  | 1 KB
ProjectionDrift      | 4.0 µs      | 251,189     | 500        | 0  | 32 KB
SentinelQuery        | 55 ns p50   | 392,952     | 175        | 0  | 155 KB

ACCEPTANCE: PASS — all detectors meet criteria.

Test plan

• 18 unit tests pass (cargo test -p ruvector-semantic-drift)
• Benchmark binary builds and runs to ACCEPTANCE PASS (cargo run --release -p ruvector-semantic-drift --bin benchmark)
• Zero false positives on 5,000-vector IID baseline for all three detectors
• All detectors detect 8σ mean shift within MAX_LAG=2,000 vectors

Key design notes

Reference window freeze: All detectors only advance the reference window when the current window is stable (score ≤ threshold). Without this, drifted windows become the new reference, scores drop to near-zero, and is_drifted() flickers rather than latching.

Mean-all-distance over Jaccard top-k: The sentinel metric uses mean squared L2 distance to all snapshot vectors instead of Jaccard overlap of top-k neighbours. High-dimensional Gaussian data has a crowding effect (many near-equidistant neighbours) that makes Jaccard top-k rank-ordering noise-sensitive. Mean-all-distance has CoV ≈ 1/√n ≈ 2% for n=300.

Dimension-agnostic thresholds: Centroid distance is normalised by √dim, making threshold=0.10 work across any embedding dimension for unit-variance models.

---

Generated by Claude Code