feat: agent-proposed test cases from a captured trace

[JabbaKadabra]

Summary

Adds Generate tests to the trace detail panel: an agent reads a trace's whole conversation and
proposes the test cases actually worth building — the turns where the agent decided something,
not every turn — as GREEN promotions (lock in what it did) or RED corrections (assert what it
should have done), each with its reasoning. Candidates are reviewed beside the transcript, expected
output is editable, and a plain-language instruction bar refines the round. Turns it passed over
are listed with the reason, so its judgement is auditable rather than opaque.

This replaces the single-click Add test promote dialog, which did a strict subset of the same
job. Traces can still be added to a suite without the agent via Add from traces on the Test
Suites page — that route stays un-gated. Generation itself is Enterprise (TestCaseSynthesis).

What's in it

The unpassable-correction trap. A correction built on the last call of a tool loop can never
pass: that call's input already contains the tool calls and their results, so the only thing
left to grade is the closing summary. Such proposals are flagged, left unchecked, and the agent is
instructed to target the call whose own response holds the decision. Tracey's test-driven
improvement playbook now routes through propose_test_cases for exactly this reason — it decides
whether the resulting regression test can ever go green — and a prompt-lab scenario probes the live
model to confirm the wording holds.

Evaluator suggestion. When the destination suite's evaluators cannot score what was proposed,
the panel offers an agentic judge: added to that suite (blast radius stated up front, since a
suite's evaluators score every case in it) or carried by a new suite instead. Each of the three
answers names its consequence.

Exact Match now compares tool calls. An expected output that is a tool call carries no text
content, so the previous content-only comparison scored any tool-free response as a pass — a
tool-call expectation asserted nothing. Calls are now compared as an unordered multiset (parallel
calls have no meaningful order), ignoring the provider-generated call id and comparing arguments as
canonical JSON so {"amount": 40} matches {"amount": 40.0}. Existing cases with tool-call
expectations are judged for real from this release, so some will correctly start failing. A/B
validation is unaffected; historical pass rates are not recomputed, so trend lines may show a step.

Sampling parameters reach the provider. Reasoning effort was mapped onto the outgoing request
and then silently dropped before it left the process — the playground's control did nothing, with
no error. Now sent for real. (n is still dropped; tracked separately.)

~3x faster generation. The panel spent its wait on hidden reasoning it did not need — several
thousand thinking tokens for a few hundred tokens of answer, 25–44s for a four-call conversation.
It now asks the model not to reason (8–13s for the same proposals) and retries without the setting
on models that lack it, so the worst case is the old speed rather than an error. The candidate
column names what it is doing and shows a running clock while a round runs.

Layers touched

• Domain — ToolRequestMatch, ExactMatchEvaluator
• Application — TestCase/ synthesis service, budgeted ConversationTranscript,
  ProposalValidator (proposals validated against the real conversation)
• Api — POST /api/agent-calls/{id}/test-case-proposals
• Infrastructure — ModelClient / ChatClientExtensions sampling parameters
• Licensing — TestCaseSynthesis (Enterprise)
• Frontend — trace-detail/synthesis/*, SynthesizeTestsModal, Tracey propose_test_cases
  tool + ProposedCasesToolUI, Toast/Radio primitives

Docs

docs/optimization-loop.md, docs/licensing.md, docs/architecture.md, docs/i18n.md,
docs/commands.md, frontend/docs/BEST_PRACTICES.md, frontend/docs/TRACEY.md; manual pages for
test suites, evaluators, traces and Tracey, with two new kiosk screenshots of the panel (including
the unpassable warning). CHANGELOG [Unreleased] covers all of the above.

Also here: i18n:translate now falls back to the repo-root KIOSK_LLM_* config, and the five
catalogs are translated (148 strings; French "outil" hand-corrected to the catalog's existing
"appel de Tool").

Tests

Backend unit + API tests for synthesis, transcript budgeting, proposal validation, tool-request
matching and the sampling parameters; frontend specs for the modal, selection rules and Toast; an
e2e spec generating test cases from a trace and adding them to a suite, plus a licensing-gate spec.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SivSrf9XfAP6Ds1jMCMukB