Skip to content

feat: retrieval-quality eval harness (recall@10, MRR) on a labeled query set #312

Description

@DevanshuNEU

Problem

OCI has no retrieval-quality measurement. Grep across backend/ and tests/ finds no recall/MRR/precision/nDCG code, no labeled query->expected-file ground truth, and no eval harness. The public CodeContextBench was paused 2026-05-13 for positioning reasons, but an internal eval is needed: we cannot currently answer "how good is search?" with a number.

Scope (v0.1)

IN:

  • A labeled ground-truth set: 30-50 realistic queries -> expected file(s)/function(s), against repos already indexed locally (34 on disk; largest ~6k files).
  • A runner that executes each query through the live search path and computes recall@k (k=5,10), MRR, precision@k.
  • Reproducible: fixed query set + fixed repo set committed; one command prints the metrics table.

OUT:

  • Public benchmark / leaderboard / methodology paper (that is the paused CodeContextBench; this is internal-only).
  • Comparison against external baselines (raw Claude tools, plain RAG) -- a later increment.

Acceptance criteria

  • Ground-truth file (queries + expected results) committed under benchmarks/ or backend/eval/.
  • Runner computes recall@10 and MRR over the set and prints a table; deterministic.
  • README documents how to add a query and re-run.
  • One real recall@10 + MRR number recorded (the resume bullet).

Notes

  • This is design-labeled: run /oci-design first (methodology choices: which search version v2 vs v3 is under test, how ground truth is labeled, k values).
  • Evidence of the gap: no matches for recall/mrr/precision/ndcg/ground_truth in backend.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions