feat(qa): promote QA roles on the evidence gate, drop manual opt-in - #317
Conversation
QA agents (qa_reviewer, qa_fixer) on a direct-API provider were gated by a DOUBLE gate: the AUTO_CODE_QA_DIRECT_RUNTIME opt-in AND the direct-API promotion gate. The opt-in existed mainly because qa_fixer's runtime path lacked recovery parity; with that closed, the opt-in is redundant. Drop it. A portable QA agent on a non-Claude provider now routes through the runtime layer whenever the direct-API promotion gate is satisfied — the same evidence the coder uses. The gate stays autonomy-aware (resolve_autonomy_settings): it only allows under AUTO_CODE_AUTONOMY=safe/ bold + recorded evidence, so at the default `claude` level QA stays on the Claude SDK even with a passing history (no opt-in hole). - Remove QA_DIRECT_RUNTIME_ENV, _qa_direct_runtime_opt_in, and the opt-in condition; the gate-denied message now carries the promotion-gate status/reason/missing + the AutonomyPolicy snapshot. - Tests: drop the opt-in from the promotion paths; add guards that `safe` without evidence still fails, and that recorded evidence WITHOUT safe stays blocked. 17 passed in test_qa_runtime_integration.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Oleg Miagkov <mrobenner@gmail.com>
|
Warning Review limit reached
More reviews will be available in 41 minutes and 22 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Live-build finding (2026-06-13, spec 901): with QA roles routed to a direct provider (qa_reviewer/qa_fixer on openai via the promotion gate, now that #317 dropped the opt-in), the QA runtime session was built with the QA phase-model DEFAULT — which phase_config resolves to a claude-* id — and OpenAI 404'd on claude-sonnet-4-5-20250929. This is the QA-path analog of the coder fix in #337: once routing correctly sends QA to the direct provider, the Claude-centric phase default leaks straight into the session. New ProviderConfig.coherent_session_model() returns the provider's own configured model when the requested model is a Claude-family id and the provider is non-Claude; otherwise it passes the request through unchanged. Both QA runtime shims (run_qa_reviewer_runtime_session, _build_qa_fixer_runtime_session) now route their model through it before create_session. The qa_fixer recovery loop already uses get_fallback_model() (#316 parity), so once the initial model is the provider's, the fallback chain (gpt-5.2 -> gpt-5 -> ...) stays coherent. Tests: 5 helper cases (claude->provider, explicit preserved, None, claude-provider unchanged, no-configured-model keeps request) + existing QA runtime / factory suites. 85 passed. Signed-off-by: Oleg Miagkov <mrobenner@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Context
QA agents (
qa_reviewer,qa_fixer) on a direct-API provider were behind a double gate: theAUTO_CODE_QA_DIRECT_RUNTIMEopt-in and the direct-API promotion gate. The opt-in existed mainly becauseqa_fixer's runtime path lacked recovery parity — closed in #316. With parity in place, the manual opt-in is redundant.This is the second half of "QA roles promote on direct providers without the manual opt-in."
What changes (
apps/backend/agents/runtime/qa_phase_routing.py)QA_DIRECT_RUNTIME_ENV,_qa_direct_runtime_opt_in, and the opt-in condition.resolve_autonomy_settings): it only allows underAUTO_CODE_AUTONOMY=safe/bold+ recorded evidence. At the defaultclaudelevel QA stays on the Claude SDK even with a passing history — no opt-in hole.status/reason/missing+ theAutonomyPolicysnapshot.Tests (
tests/test_qa_runtime_integration.py, 17 passed)safewithout evidence still fails fast; recorded evidence withoutsafe(claude level) stays blocked.On "per-role evidence"
The promotion gate is per-provider and the provider e2e suite exercises the exact runtime mechanism the QA roles use (generic_edit + recovery + transaction). So the coder's recorded evidence already covers the runtime capability QA depends on. A dedicated per-role QA smoke (separate evidence stream) is a possible future refinement, noted but not required for promotion.
🤖 Generated with Claude Code