Skip to content

fix(qa): persist direct-API qa_fixer status via a qa_status.json artifact - #342

Merged
OBenner merged 1 commit into
developfrom
claude/priceless-banach-3935b8
Jun 18, 2026
Merged

fix(qa): persist direct-API qa_fixer status via a qa_status.json artifact#342
OBenner merged 1 commit into
developfrom
claude/priceless-banach-3935b8

Conversation

@OBenner

@OBenner OBenner commented Jun 13, 2026

Copy link
Copy Markdown
Owner

Problem

On a direct-API provider (e.g. OpenAI via the generic_edit runtime), the qa_fixer can't record its status, so a REJECTED → fix → re-QA loop stalls. prompts/qa_fixer.md tells the model to write qa_signoff into implementation_plan.json using Claude's Write/Edit tools and absolute paths — but in generic_edit those tools don't exist (the real tools are write_file/replace_text) and resolve_workspace_path rejects absolute paths. The result: ready_for_qa_revalidation is never set, is_fixes_applied stays false, and the fixer reports error even after applying real fixes.

Fix

Give the direct-API fixer a file-based status contract it can actually satisfy, mirroring the file-based verdict the Claude path already relies on:

  1. Prompt overridebuild_qa_fixer_prompt gains runtime_status_relspec. When set, a generic_edit addendum overrides the Claude Write/Edit/absolute-path guidance: edit with write_file using workspace-relative paths, and record completion by writing a workspace-relative qa_status.json (status="fixes_applied", ready_for_qa_revalidation, fix_session, fixes_summary), then finish. The Claude path is unchanged (param defaults to None).
  2. Deterministic mergerun_qa_fixer_via_runtime computes the spec dir relative to the project root and, when is_fixes_applied is false, folds <spec_dir>/qa_status.json into qa_signoff (load_/save_implementation_plan), then re-checks. Pre-existing signoff fields are preserved.
  3. Same-loop async closerun_qa_fixer_runtime_session async-closes each per-attempt provider session in a finally (rebuilt per attempt by the recovery loop). Adds OpenAICompatibleSession.aclose() to await the underlying AsyncOpenAI connection-pool close — the sync close() only drops references, leaking sockets.

Tests

tests/test_qa_fixer_runtime.py18 passed (10 existing + 8 new), ruff clean:

apps/backend/.venv/bin/python -m pytest tests/test_qa_fixer_runtime.py -p no:cacheprovider -q

New coverage: scripted runtime session writes qa_status.json → returns "fixed" with the artifact merged into qa_signoff; no-artifact stays "error" (loop must not falsely pass); the prompt-addendum content; per-attempt async close; and the OpenAICompatibleSession.aclose() primitive.

Follow-up (not in this PR)

The qa_reviewer runtime path (run_qa_reviewer_via_runtime) has the same latent status-persistence bug — it also can't write qa_signoff on a direct-API provider. Its verdict is richer (approved/rejected + issues_found), so it needs the same pattern with an extended artifact schema. Tracked separately to keep this change focused.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/priceless-banach-3935b8

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added area/backend bug Something isn't working size/M labels Jun 13, 2026
Comment thread apps/backend/qa/fixer.py
``fixes_applied`` artifact was merged and persisted.
"""
# Lazy import to avoid the qa.criteria -> ... -> qa.reviewer/fixer cycle.
from .criteria import load_implementation_plan, save_implementation_plan
…fact

The generic_edit qa_fixer could not record "fixes_applied" on a non-Claude
provider, so a REJECTED -> fix -> re-QA loop on e.g. OpenAI stalled. The
qa_fixer.md prompt tells the model to write qa_signoff into
implementation_plan.json with Claude's Write/Edit tools and absolute paths,
but on the generic_edit runtime those tools don't exist (the real tools are
write_file/replace_text) and resolve_workspace_path rejects absolute paths,
so ready_for_qa_revalidation was never set and is_fixes_applied stayed false.

- build_qa_fixer_prompt gains runtime_status_relspec; when set it appends a
  generic_edit addendum that OVERRIDES the Claude Write/Edit/absolute-path
  guidance: edit with write_file using workspace-relative paths, and record
  completion by writing a workspace-relative qa_status.json
  (status=fixes_applied, ready_for_qa_revalidation, fix_session,
  fixes_summary) then calling finish.
- run_qa_fixer_via_runtime computes the relspec via spec_dir.relative_to(
  project_dir) and, when is_fixes_applied is false, deterministically folds
  <spec_dir>/qa_status.json into qa_signoff (load/save_implementation_plan)
  before re-checking — matching the Claude path's file-based verdict.
- run_qa_fixer_runtime_session async-closes each per-attempt provider session
  in the same event loop; add OpenAICompatibleSession.aclose() to await the
  AsyncOpenAI connection-pool close so rebuilt-per-attempt sessions don't leak
  sockets.

Add regression tests in tests/test_qa_fixer_runtime.py: a scripted runtime
session that writes qa_status.json -> "fixed" with the artifact merged into
qa_signoff, the prompt-addendum content, per-attempt async close, and the
OpenAICompatibleSession.aclose() primitive.

Signed-off-by: Oleg Miagkov <mrobenner@gmail.com>
@OBenner
OBenner force-pushed the claude/priceless-banach-3935b8 branch from 3e98c5c to 664b8a8 Compare June 18, 2026 09:24
@sonarqubecloud

Copy link
Copy Markdown

@OBenner
OBenner merged commit 0f0da8b into develop Jun 18, 2026
18 of 19 checks passed
@OBenner
OBenner deleted the claude/priceless-banach-3935b8 branch June 18, 2026 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/backend bug Something isn't working size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants