Skip to content

feat(code-review): implement confidence scoring and --threshold flag - #82794

Open
hulincup wants to merge 8 commits into
anthropics:mainfrom
hulincup:feat/code-review-confidence-scoring
Open

feat(code-review): implement confidence scoring and --threshold flag#82794
hulincup wants to merge 8 commits into
anthropics:mainfrom
hulincup:feat/code-review-confidence-scoring

Conversation

@hulincup

Copy link
Copy Markdown

Summary

Reconciles README↔command drift in the code-review plugin: the documented 0–100 confidence scoring was never implemented (the command used binary validation). This PR implements scoring as a single validate-and-score pass that preserves the existing truth-check while making the documented threshold actually configurable via --threshold.

Relationship to existing PR #79150

Open PR #79150 ("docs: align code-review README with the current validation-based command") identifies the same drift but takes the opposite approach: it removes the scoring/threshold documentation from the README to match the command's current validation-only behavior.

This PR instead implements the scoring in the command. I believe implement > remove because:

  • The README's scoring + threshold was an intentional design (filter false positives by severity), not dead code. Removing it deletes a useful feature; implementing it delivers a working, tunable severity filter.
  • The default threshold of 80 (only definite bugs/violations ship) matches the command's existing "HIGH SIGNAL only" stance — so default behavior gets stricter and more aligned with the plugin's stated purpose, not noisier.
  • It adds zero additional subagent calls versus current behavior (step 5 is refactored from validate-only to validate+score in the same single pass; no second scoring pass).
  • It makes the documented threshold a real --threshold flag (the README previously told users to edit a "Filter out any issues with a score less than 80." line that does not exist in the command — this PR replaces that dead instruction with a working flag).

I'm opening this as an alternative path for the maintainers; happy to close it if #79150's remove approach is preferred.

Changes

  • commands/code-review.md step 5: each issue subagent now both verifies the issue is real and assigns a confidence score, returning a structured result (verdict, score, evidence, rule_citation). Preserves the existing false-positive list and the Opus/Sonnet agent split.
  • commands/code-review.md step 6: filters on verdict == real AND score >= threshold (default 80) instead of binary validation.
  • commands/code-review.md step 7: terminal output now shows (score N) per issue so a human reviewer can re-prioritize. Scores are not posted to the PR.
  • New --threshold <n> flag (integer 0–100, default 80) with fail-fast validation. Composes with --comment.
  • README.md: scoring section now documents the actual 0/25/50/75/100 rubric; Configuration section uses --threshold instead of telling users to edit the command file.
  • plugin.json: 1.0.0 → 1.1.0 (additive feature, semver minor).

Design rationale

The README describes scoring (rate confidence 0–100); the command implemented validation (binary real/not-real). These are different ideas — a confident-but-wrong issue could score 100 under pure scoring. This PR merges them into one pass: the subagent verifies truth and rates severity. This preserves the plugin's false-positive filtering while delivering the documented scoring.

The score is a discrete enum {0, 25, 50, 75, 100} (not a continuous integer) so two agents can't diverge at the threshold boundary — at default threshold 80, only score=100 (definite bugs/violations) ships. The threshold is a continuous integer 0–100 so users can tune the floor.

No new subagent calls versus prior behavior (single pass per issue, same as today's validation pass). No new tools, no new MCP dependencies.

Out of scope (separate PRs)

  • README drift on agent count ("5 parallel Sonnet agents") and the absent "history analyzer" agent.
  • --pr <number> flag for reviewing a PR without checking out its branch.
  • Fallback for mcp__github_inline_comment__create_inline_comment to gh pr comment.

Test plan

  • Self-consistency: command step numbering 1–9 intact; every step referenced in 7/9 still exists.
  • Rubric↔step alignment: README rubric table matches the scores step 5 assigns.
  • Flag-parsing trace: --threshold / --comment combinations parse unambiguously; out-of-range (150, -5) and non-integer (abc) values rejected.
  • Dry-run trace: mentally traced the prompt against PR Fix examples/gateway/aws/setup.sh aborting on stock macOS bash 3.2 #82320 (small bash portability fix); scoring is predictable (a trivial nit → score 25, dropped at default threshold; a real syntax error → score 100, ships).
  • (Maintainer) Run /code-review against a sample PR on a fork and confirm terminal output shows (score N) and the filter threshold behaves.

🤖 Generated with Claude Code

hulincup and others added 8 commits July 31, 2026 12:43
Plumbing only; the threshold is not consumed until step 5/6 are
rewritten to score issues. Adds arg docs to the command body.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces binary validation with a single validate-and-score pass:
each issue subagent now returns verdict + 0-100 score + evidence +
rule_citation. Step 6 filters on verdict=real AND score>=threshold
(default 80, via --threshold). Preserves the existing truth-check
and false-positive list. No extra subagent calls vs. prior behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Score field was 'integer 0-100' but the rubric only has 5 discrete
buckets {0,25,50,75,100} with 'assign the closest match'. Two agents
could return 75 (bucket) vs 80 (interpolated) for the same issue,
breaking threshold determinism. Make the score explicitly one of the
five buckets. Also defines score=0 for false_positive verdicts,
resolving a contract inconsistency (code-quality review Issue 2).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 7 now appends (score N) to each listed issue so a human
reviewer can re-prioritize. Scores are not posted to the PR; step 9
(inline comments) is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Example was '(score 90)' but step 5 defines score as a discrete enum
{0,25,50,75,100} and default threshold 80 filters to only score=100.
Use '(score 100)' with a score-100-flavored issue so the example is
internally coherent end-to-end (code-quality review Issue 1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Rubric table now matches the 0/25/50/75/100 buckets the command
  assigns, with evidence and verdict semantics.
- Configuration section uses the --threshold flag instead of telling
  users to edit the command file.
- Usage/options block documents --threshold.

Scope: scoring + threshold only. README agent-count drift
('5 parallel Sonnet agents', 'history analyzer') is intentionally
left for a separate PR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README said 'scored 0-100' and '(0-100)' for the score, but the
command defines score as a discrete enum {0,25,50,75,100}. A reader
could believe 80/90 is assignable and be confused when a 75 is
filtered at default threshold 80. The threshold (integer 0-100) is
unaffected. Aligns README score phrasing with the command contract
(code-quality review Issue 5).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Additive feature: --threshold flag + 0-100 confidence scoring.
Semver minor bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant