From 5ca702efa83ab90aa065caa09f0430587ef0613b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 16:01:08 +0000 Subject: [PATCH 1/2] docs: record the RLVR training-loop insight in the LLM section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a "The deploy loop is a training loop (RLVR)" subsection to research-landscape-2026.md: the fix-loop's proposal->verdict pairs are free labelled examples, so deployment is data generation. Captures why a sound deterministic verifier is a cleaner reward than RLHF's learned reward model; the free bootstrap from deterministic fixes + rejection-sampling SFT (no RL infra needed); and the load-bearing trap — reward hacking / Goodhart (checker-green != behaviour-preserved), so the metamorphic harness becomes part of the reward function, with the verifier setting both the ceiling and the floor. Re-confirms fix > spec from the training side (a fix has a verifiable reward; a mined spec does not). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rg8kSk1YT14x7A1vo5zgED --- docs/notes/research-landscape-2026.md | 39 +++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/notes/research-landscape-2026.md b/docs/notes/research-landscape-2026.md index 765ea5cf..5ded10ed 100644 --- a/docs/notes/research-landscape-2026.md +++ b/docs/notes/research-landscape-2026.md @@ -119,6 +119,45 @@ accept/reject. Honest caveat: **green ≠ behavior-preserving**, so the loop nee a **metamorphic conformance harness** (the doc's StaAgent / Statfier line) as the check, not human trust. +### The deploy loop is a training loop (RLVR) — and its one trap + +The fix-loop has a second payoff that falls out for free: **every +`proposal → verdict` pair is a labelled training example**, so the deployment +loop *is* a data-generation loop. This is **RL from Verifiable Rewards (RLVR)** — +the regime that carried the recent frontier math/code gains — and it fits here +unusually well: + +- The reward comes from a **deterministic, sound verifier**, not a learned reward + model. RLHF's reward model is itself a fallible LLM (noisy, gameable); our + checker + compile + tests is ground truth, so the signal is clean — precisely + the verifiable-domain regime where RLVR actually works rather than just markets. +- **No human-labelled dataset** — the checker manufactures the labels on real code. +- **Bootstrap is free**: the deterministic fixes (the mechanical ~80%) are + *guaranteed-correct demonstrations* to SFT on; the LLM takes the non-local ~20%, + the verifier filters. Flywheel: deploy → propose → verify → keep winners → train + → better proposals. +- Cheapest stable form needs no RL infra — **rejection sampling**: draw N fixes, + keep the ones that pass checker + compile + tests, fine-tune on the winners. + GRPO/PPO with the verifier as reward is a later, sample-efficiency move. + +**The trap — and it is the same caveat as everywhere above.** "Checker went +quiet" ≠ "bug fixed correctly". Optimise literally for that and the model learns +**reward hacking / Goodhart**: delete the resource, suppress the diagnostic, +weaken the code, remove the failing test — Own.NET green, behaviour broken. This +is RLVR-for-code's documented failure mode (the model hacks the test suite, not +the task). So the **metamorphic harness stops being only a deployment check and +becomes part of the reward function**: the reward must be *checker-green AND +behaviour-preserved* (metamorphic invariants / behaviour-diff / held-out tests), +or you train a checker-gamer. The verifier sets both the **ceiling** (the model +cannot exceed what the verifier can check — it learns the blind spots as exploits) +and the **floor** (without behaviour checks it degrades). + +This re-confirms **fix > spec** from the training side too: a **fix** has a clean +verifiable reward (re-run checker + tests = the label); a **mined spec** has none +— its correctness is not locally checkable (trusted base), so there is nothing +sound to reward beyond consistency. The role worth training is the one we already +chose — the **refactor assistant, not the truth generator**. + ## Concrete backlog this implies (recorded, NOT scheduled) Ordered by leverage; each annotated with where it attaches and its real cost. From 2fcd79f49b1aa1f8514d96c1bc469f06a2cf5a8a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 16:08:00 +0000 Subject: [PATCH 2/2] docs: consistent American spelling in the RLVR subsection The new subsection mixed British forms (labelled / behaviour / Optimise) with the file's existing American variant (labeled / behavior); standardize the subsection to American. Addresses the CodeRabbit/LanguageTool consistency nit on #44. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rg8kSk1YT14x7A1vo5zgED --- docs/notes/research-landscape-2026.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/notes/research-landscape-2026.md b/docs/notes/research-landscape-2026.md index 5ded10ed..b0b4efb8 100644 --- a/docs/notes/research-landscape-2026.md +++ b/docs/notes/research-landscape-2026.md @@ -122,7 +122,7 @@ check, not human trust. ### The deploy loop is a training loop (RLVR) — and its one trap The fix-loop has a second payoff that falls out for free: **every -`proposal → verdict` pair is a labelled training example**, so the deployment +`proposal → verdict` pair is a labeled training example**, so the deployment loop *is* a data-generation loop. This is **RL from Verifiable Rewards (RLVR)** — the regime that carried the recent frontier math/code gains — and it fits here unusually well: @@ -131,7 +131,7 @@ unusually well: model. RLHF's reward model is itself a fallible LLM (noisy, gameable); our checker + compile + tests is ground truth, so the signal is clean — precisely the verifiable-domain regime where RLVR actually works rather than just markets. -- **No human-labelled dataset** — the checker manufactures the labels on real code. +- **No human-labeled dataset** — the checker manufactures the labels on real code. - **Bootstrap is free**: the deterministic fixes (the mechanical ~80%) are *guaranteed-correct demonstrations* to SFT on; the LLM takes the non-local ~20%, the verifier filters. Flywheel: deploy → propose → verify → keep winners → train @@ -141,16 +141,16 @@ unusually well: GRPO/PPO with the verifier as reward is a later, sample-efficiency move. **The trap — and it is the same caveat as everywhere above.** "Checker went -quiet" ≠ "bug fixed correctly". Optimise literally for that and the model learns +quiet" ≠ "bug fixed correctly". Optimize literally for that and the model learns **reward hacking / Goodhart**: delete the resource, suppress the diagnostic, -weaken the code, remove the failing test — Own.NET green, behaviour broken. This +weaken the code, remove the failing test — Own.NET green, behavior broken. This is RLVR-for-code's documented failure mode (the model hacks the test suite, not the task). So the **metamorphic harness stops being only a deployment check and becomes part of the reward function**: the reward must be *checker-green AND -behaviour-preserved* (metamorphic invariants / behaviour-diff / held-out tests), +behavior-preserved* (metamorphic invariants / behavior-diff / held-out tests), or you train a checker-gamer. The verifier sets both the **ceiling** (the model cannot exceed what the verifier can check — it learns the blind spots as exploits) -and the **floor** (without behaviour checks it degrades). +and the **floor** (without behavior checks it degrades). This re-confirms **fix > spec** from the training side too: a **fix** has a clean verifiable reward (re-run checker + tests = the label); a **mined spec** has none