docs(design): propose hotspot split M3 — automation

[bootjp]

Design-doc-only PR. No implementation code. Per the repo's design-doc-first workflow, this lands the M3 proposal for review before any implementation.

New file: docs/design/2026_06_11_proposed_hotspot_split_milestone3_automation.md (Status: Proposed, Author: bootjp, Date: 2026-06-11).

Summary

Milestone 3 of the hotspot shard split design: automation — auto-detection + an auto-split scheduler. M1 (durable versioned route catalog, manual same-group SplitRange, route watcher) has shipped; M2 (#945) proposes the cross-group migration plane. M3 composes with M2 but delivers first value standalone: auto-detection + same-group auto-split (the M1 capability) work without M2; cross-group target selection slots in behind the same SplitRange interface once M2 lands.

It closes the gap the parent doc §1 calls out: distribution/engine.go's RecordAccess/threshold-splitRange exists but is wired into nothing (only tests call it; verified by grep) and is midpoint-only + mutates the engine directly (bypassing the catalog/watcher).

Key design points

• Detection signal — reuse the keyviz sampler, retire the engine counters. The keyviz MemSampler is already wired into the exact request-path functions the parent doc names (groupMutations→observeMutation, observeRead in kv/sharded_coordinator.go), is allocation-free on the hot path, splits reads vs writes, is windowed, has sub-range + Top-K split-key evidence, and bounds cardinality. Driving detection off it avoids a second parallel pipeline. The engine's dead RecordAccess/splitRange/Route.Load path is removed in M3-PR1. M3 adds zero hot-path cost — it consumes already-flushed windows off-path.
• Aggregation — leader-local, no new RPC. The detector runs on the default-group leader and reads its own MemSampler.Snapshot for routes it leads (the leader serves the load it would split on). No ReportAccess RPC and no RPC piggybacking; cluster fan-out stays an admin-UI concern and can slot behind the detector interface later if needed.
• Detector — score = write_ops*Ww + read_ops*Wr (4/1), 50k ops/min threshold, 3-consecutive-window up-side hysteresis + 0.8× down-side band, per-route 10-min cooldown (monotonic deadline, not wall clock), split-key from observed distribution (sub-range p50 / single-key isolation / decline when no evidence — never blind midpoint), guardrails: min route span, max routes, per-cycle split budget.
• Scheduler — invokes the existing proto.Distribution.SplitRange RPC with expected_catalog_version CAS; all catalog mutations go through SplitRange (repo convention). Same-group today; once M2 lands the least-loaded-group target policy flips the target_group_id field on the same RPC.
• Safety/ops — default OFF behind --autoSplit; runtime kill switch (atomic.Bool); fixed-enum metrics (no per-route/key labels); stable slog keys; detector state is leader-local and resets on election (re-earns confidence; cooldown re-seeded from parent_route_id lineage) — stated explicitly so non-replication isn't mistaken for an oversight.
• PR breakdown — M3-PR1 (retire engine path + metric scaffolding), M3-PR2 (aggregation + pure detector, observe-only), M3-PR3 (scheduler + flag + kill switch), each independently shippable; M3-PR4 enables cross-group target selection post-M2.

Every current-state claim cites concrete file:line evidence, verified before writing.

Open questions

1. OQ-1 — Fully delete Route.Load/Engine.Stats, or keep Stats for diagnostics?
2. OQ-2 — Down-side hysteresis: hold the counter in the [0.8×, 1.0×) band, or decay it?
3. OQ-3 — No sub-range/Top-K evidence: decline to split (proposed), or allow a min/max-observed-key midpoint?
4. OQ-4 — minRouteSpan without per-route key counts: approximate from sampler evidence, or add a cheap key-count estimate?
5. OQ-5 — Leader-local scoring sufficient, or is there a heavy-follower-read workload needing cluster fan-out?
6. OQ-6 — Post-M2: enable cross-group target automatically, or gate behind --autoSplitCrossGroup?
7. OQ-7 — Durable cooldown across elections: is parent_route_id + a catalog split timestamp enough, or do we need durable last_split_at?
8. OQ-8 — Express threshold in ops/min (proposed) or ops-per-Step to avoid coupling sensitivity to --keyvizStep?

Test plan

Doc-only — no Go tests run. Markdown verified clean; all cited file:line references confirmed against main. The doc's own §8.2 lays out the implementation-phase test strategy (table-driven detector unit tests + pgregory.net/rapid invariants, detector→SplitRange integration in the 3-node demo, Jepsen hotspot workload deferred to M4 per the parent doc).