feat(code-review): implement confidence scoring and --threshold flag - #82794
Open
hulincup wants to merge 8 commits into
Open
feat(code-review): implement confidence scoring and --threshold flag#82794hulincup wants to merge 8 commits into
hulincup wants to merge 8 commits into
Conversation
Plumbing only; the threshold is not consumed until step 5/6 are rewritten to score issues. Adds arg docs to the command body. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces binary validation with a single validate-and-score pass: each issue subagent now returns verdict + 0-100 score + evidence + rule_citation. Step 6 filters on verdict=real AND score>=threshold (default 80, via --threshold). Preserves the existing truth-check and false-positive list. No extra subagent calls vs. prior behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Score field was 'integer 0-100' but the rubric only has 5 discrete
buckets {0,25,50,75,100} with 'assign the closest match'. Two agents
could return 75 (bucket) vs 80 (interpolated) for the same issue,
breaking threshold determinism. Make the score explicitly one of the
five buckets. Also defines score=0 for false_positive verdicts,
resolving a contract inconsistency (code-quality review Issue 2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 7 now appends (score N) to each listed issue so a human reviewer can re-prioritize. Scores are not posted to the PR; step 9 (inline comments) is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Example was '(score 90)' but step 5 defines score as a discrete enum
{0,25,50,75,100} and default threshold 80 filters to only score=100.
Use '(score 100)' with a score-100-flavored issue so the example is
internally coherent end-to-end (code-quality review Issue 1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Rubric table now matches the 0/25/50/75/100 buckets the command
assigns, with evidence and verdict semantics.
- Configuration section uses the --threshold flag instead of telling
users to edit the command file.
- Usage/options block documents --threshold.
Scope: scoring + threshold only. README agent-count drift
('5 parallel Sonnet agents', 'history analyzer') is intentionally
left for a separate PR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README said 'scored 0-100' and '(0-100)' for the score, but the
command defines score as a discrete enum {0,25,50,75,100}. A reader
could believe 80/90 is assignable and be confused when a 75 is
filtered at default threshold 80. The threshold (integer 0-100) is
unaffected. Aligns README score phrasing with the command contract
(code-quality review Issue 5).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Additive feature: --threshold flag + 0-100 confidence scoring. Semver minor bump. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reconciles README↔command drift in the
code-reviewplugin: the documented 0–100 confidence scoring was never implemented (the command used binary validation). This PR implements scoring as a single validate-and-score pass that preserves the existing truth-check while making the documented threshold actually configurable via--threshold.Relationship to existing PR #79150
Open PR #79150 ("docs: align code-review README with the current validation-based command") identifies the same drift but takes the opposite approach: it removes the scoring/threshold documentation from the README to match the command's current validation-only behavior.
This PR instead implements the scoring in the command. I believe implement > remove because:
--thresholdflag (the README previously told users to edit a "Filter out any issues with a score less than 80." line that does not exist in the command — this PR replaces that dead instruction with a working flag).I'm opening this as an alternative path for the maintainers; happy to close it if #79150's remove approach is preferred.
Changes
commands/code-review.mdstep 5: each issue subagent now both verifies the issue is real and assigns a confidence score, returning a structured result (verdict,score,evidence,rule_citation). Preserves the existing false-positive list and the Opus/Sonnet agent split.commands/code-review.mdstep 6: filters onverdict == real AND score >= threshold(default 80) instead of binary validation.commands/code-review.mdstep 7: terminal output now shows(score N)per issue so a human reviewer can re-prioritize. Scores are not posted to the PR.--threshold <n>flag (integer 0–100, default 80) with fail-fast validation. Composes with--comment.README.md: scoring section now documents the actual 0/25/50/75/100 rubric; Configuration section uses--thresholdinstead of telling users to edit the command file.plugin.json: 1.0.0 → 1.1.0 (additive feature, semver minor).Design rationale
The README describes scoring (rate confidence 0–100); the command implemented validation (binary real/not-real). These are different ideas — a confident-but-wrong issue could score 100 under pure scoring. This PR merges them into one pass: the subagent verifies truth and rates severity. This preserves the plugin's false-positive filtering while delivering the documented scoring.
The score is a discrete enum
{0, 25, 50, 75, 100}(not a continuous integer) so two agents can't diverge at the threshold boundary — at default threshold 80, onlyscore=100(definite bugs/violations) ships. The threshold is a continuous integer 0–100 so users can tune the floor.No new subagent calls versus prior behavior (single pass per issue, same as today's validation pass). No new tools, no new MCP dependencies.
Out of scope (separate PRs)
--pr <number>flag for reviewing a PR without checking out its branch.mcp__github_inline_comment__create_inline_commenttogh pr comment.Test plan
--threshold/--commentcombinations parse unambiguously; out-of-range (150,-5) and non-integer (abc) values rejected./code-reviewagainst a sample PR on a fork and confirm terminal output shows(score N)and the filter threshold behaves.🤖 Generated with Claude Code