Problem
OCI has no retrieval-quality measurement. Grep across backend/ and tests/ finds no recall/MRR/precision/nDCG code, no labeled query->expected-file ground truth, and no eval harness. The public CodeContextBench was paused 2026-05-13 for positioning reasons, but an internal eval is needed: we cannot currently answer "how good is search?" with a number.
Scope (v0.1)
IN:
- A labeled ground-truth set: 30-50 realistic queries -> expected file(s)/function(s), against repos already indexed locally (34 on disk; largest ~6k files).
- A runner that executes each query through the live search path and computes recall@k (k=5,10), MRR, precision@k.
- Reproducible: fixed query set + fixed repo set committed; one command prints the metrics table.
OUT:
- Public benchmark / leaderboard / methodology paper (that is the paused CodeContextBench; this is internal-only).
- Comparison against external baselines (raw Claude tools, plain RAG) -- a later increment.
Acceptance criteria
Notes
- This is design-labeled: run
/oci-design first (methodology choices: which search version v2 vs v3 is under test, how ground truth is labeled, k values).
- Evidence of the gap: no matches for recall/mrr/precision/ndcg/ground_truth in backend.
Problem
OCI has no retrieval-quality measurement. Grep across
backend/andtests/finds no recall/MRR/precision/nDCG code, no labeled query->expected-file ground truth, and no eval harness. The public CodeContextBench was paused 2026-05-13 for positioning reasons, but an internal eval is needed: we cannot currently answer "how good is search?" with a number.Scope (v0.1)
IN:
OUT:
Acceptance criteria
benchmarks/orbackend/eval/.Notes
/oci-designfirst (methodology choices: which search version v2 vs v3 is under test, how ground truth is labeled, k values).