test: run repository-mutating tests against per-test isolated repositories - #231
Merged
Conversation
…ories Tests that mutate whole-repository state (locales, push, sync, status) previously ran serially against the shared repository. They now create a throwaway repository per test via an it.scoped repo override, letting the whole suite run concurrently. Tests that do not touch repository state, like --help and error-path tests, keep the shared repository. The isolatedRepo fixture takes no other fixtures as dependencies because vitest resolves it.scoped override dependencies only against other scoped fixtures, silently passing undefined for anything from test.extend. Also flattens the vitest config to a single test config now that the serial project is gone. This enabled the previously inert root-level typecheck option and surfaced that concurrent tests must use onTestFinished from the test context rather than the global import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n fixture
Suites opt into a throwaway repository with it.scoped({ isolateRepo: true })
instead of overriding repo with a standalone function. The repo fixture
branches on the flag and uses normal fixture dependencies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
angeloashmore
marked this pull request as ready for review
July 22, 2026 01:49
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9fbb2a0. Configure here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
angeloashmore
added a commit
that referenced
this pull request
Jul 22, 2026
Adopts main's isolateRepo fixture API from #231: drops the serial test project split, moves the exec fixture into eval/it.ts, and switches the eval repo isolation to isolateRepo: true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
angeloashmore
added a commit
that referenced
this pull request
Jul 22, 2026
Evals only shell out to git; the generic runner's options plumbing was left over from the pre-#231 CLI fixture. throwOnError surfaces setup failures that previously passed silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
angeloashmore
added a commit
that referenced
this pull request
Jul 28, 2026
* feat: add AI eval harness for the CLI Plain vitest evals that drive a real Claude Code agent (via the Agent SDK) against the built CLI in a generic, hermetic config. Adds the `agent` fixture, an `toHaveRun` matcher, an LLM-judge matcher with a never-regress ratchet, and a per-run reporter that records efficiency metrics and the agent's prismic calls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(eval): consolidate harness into one fixture file Merge the agent fixture, matchers, judge, config isolation, and result recording into eval/it.ts; delete claude.ts, report.ts, and setup.ts. Move the judge ratchet from a read-back of results.jsonl into an inline threshold literal that vitest -u rewrites like a snapshot, so results.jsonl is write-only. Trim recorded fields to effectiveness and effort (turns, tokens, duration); drop cost, which swings with provider pricing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(eval): drop explanatory comments from harness Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(eval): harden judge, run isolation, and bar ratchet - Judge uses the SDK's native structured output instead of hand-parsing JSON; score is range-checked to [0, 1] - Create a throwaway Claude config dir per agent/judge call and clean it up; set persistSession: false so runs stay ephemeral - Detect -u from vitest's update state; the worker's process.argv is empty - Fix raiseLiteral rewriting the wrong literal on the callsite line, and support inline bars - Fix the evals npm script so vitest also receives RUN_EVALS Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(eval): score by pass rate over trials with a binary judge Each eval now runs EVAL_TRIALS times (default 3) via it.for; a custom reporter prints per-eval pass rates with infra failures excluded and appends per-trial rows (pass, cost, tokens, turns, commands) to results.jsonl. The judge is binary like agent-eval's: prose criteria, pass/fail verdicts, no scores, no ratcheting bars. The full eval catalog is implemented as it.skip drafts; only the rich-text smoke eval runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): add results digest script `node --run evals:report` prints one block per eval with one line per run (pass rate, cost, mean tokens/turns/duration) and a latest-run footer with best-of and per-trial rates. Rows now carry a per-run id so the digest groups runs exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): tag results with CLI commit and cap history Rows record the git SHA that produced them, shown in the digest so rate changes point at a commit range. The file keeps the newest 100 runs; the digest shows the last 10 per eval. results.jsonl merges with the union driver so branch runs never conflict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): simplify harness The judge now calls the Messages API directly with structured outputs instead of spawning a Claude Code session. The agent fixture writes its own task.meta via the built-in task context, trial rows no longer carry a parsed trial number, and eval env vars are parsed with a zod schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): validate evals, isolate repos, and run concurrently Enable 26 validated evals with per-trial runs and skip 3 that fail consistently, with the reason noted on each. Trim duplicate evals and merge the two slice component evals into one. Each eval now gets its own repository via an isolatedRepo fixture, letting evals run 8-way concurrent without cross-test interference. Global teardown sweeps orphaned repositories created by agents (e.g. prismic init) that the harness cannot name in advance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): add happy-path evals and count only real command runs Add four evals from the Next.js happy-path docs: init with an existing repository, page types vs custom types, commit-then-push flow, and production previews after a deploy. The last two fail today and stay enabled: evals are measurements, and their pass rates track CLI gaps. Un-skip the other agent-capability evals for the same reason; only the field rename eval stays skipped since the CLI cannot rename fields. toHaveRun now evaluates each shell segment separately and ignores --help invocations, so reading help no longer counts as running a command in either direction. Scope the results digest to the last EVAL_HISTORY runs (default 10) with a per-run footer, and document the results.jsonl schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): replace exec fixture with a git-only fixture Evals only shell out to git; the generic runner's options plumbing was left over from the pre-#231 CLI fixture. throwOnError surfaces setup failures that previously passed silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(eval): drop explanatory comments from harness Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): split evals into behavior-named files under evals/ Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(test): gate evals with an explicit opt-in and drop test-mode builds Evals now require PRISMIC_ALLOW_EVALS=true and an explicit --project evals; bare vitest runs only unit tests. Tests run against the prebuilt dist (no MODE=test build), sync's poll interval is a validated PRISMIC_SYNC_POLL_MS env var, agent-created repos are cleaned up per test instead of an account-wide sweep, and dead MODE/DEV env plumbing is removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): correct results.jsonl gitattributes path and drop unused helper Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop redundant .env.test.local gitignore entry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): use toLocaleString for run dates and trim mean guard Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): type results.jsonl rows at the write site Export AgentRecord with field docs, build Row from it in the reporter, and reuse that type in the report instead of a duplicated schema comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): define results.jsonl rows as a zod schema and parse on read Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(eval): use TSDoc comments on the row schema Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(eval): judge field names in design-sensible-models bars Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(eval): model a slice from a screenshot Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(eval): record eval run at 6c4397b Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): apply sequence.concurrent at the config root Vitest silently ignores `sequence` inside a `projects` entry, so every test in a file ran serially. Root-level `sequence.concurrent` propagates to projects and restores within-file concurrency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(eval): source agent guidance from the pinned prismicio/skills SKILL.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): simplify the harness to two files and a per-run results.json Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop the union-merge rule for the removed results.jsonl Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): record filtered runs to results.local.json only Partial runs (file filters, -t, it.only) are detected by the reporter and never overwrite results.json, so it always holds one full run. Permanent disables must be it.todo, which vitest keeps distinguishable from filter skips. Also fixes the remaining bin/ mkdir race in the project fixture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): write results.json on a single line Every run rewrites most values, so a pretty-printed file only inflates PR diffs; one line keeps run-over-run changes to one changed line. Read it with jq or node --run evals:report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): use object-form fixtures and revert to vitest 4.0.17 Aligns the eval fixtures with test/it.ts and drops the vitest 4.1 builder-pattern API, shrinking the lockfile diff to the agent SDK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eval): drop the .ts import extension and restore main's tsconfig Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eval): skip Bash tool calls without a command string Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): stop forcing throwOnError in the exec fixture Tests that assert non-zero exit codes need the result object back, not a thrown error. Restores main's behavior; the eval git setup surfaces failures through its own assertions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(test): drop the sequence placement comment * fix(eval): keep results.json untouched when a trial never reached the agent A fixture failure before the prompt leaves a failed test with no recorded trial. Recording the rest as a full run would misstate pass rates, so such runs now count as partial. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: fetch the profile for telemetry only when telemetry is enabled Every CLI invocation fired a fire-and-forget /profile request even with telemetry disabled, since the call sat outside the telemetry gates. In CI this put every spawned CLI process on user-service concurrently, which starts returning 500s at roughly eight concurrent same-user requests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(eval): drop the stale Fails comments Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: run write-token tests sequentially Wroom's settings/security/token endpoint intermittently returns 500 when the same user fires concurrent write-token creates, which the concurrent test mode did across CI matrix jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Resolves:
Description
Removes the serial test project. Tests that mutate whole-repository state (locales, push, sync, status) opt into a throwaway repository per test, so the entire suite runs concurrently:
Checklist
Preview
How to QA 1
Run
node --run test. All 83 files (288 tests) run concurrently; there are no*.serial.test.tsfiles left.🤖 Generated with Claude Code
Note
Low Risk
Changes are almost entirely test and Vitest configuration; runtime CLI behavior is unchanged aside from a whitespace-only edit in
push.ts.Overview
Replaces serial e2e with per-test isolated Prismic repositories so the whole suite can run concurrently. Vitest’s separate
serialproject and all*.serial.test.tsfiles are removed; a single config enablessequence.concurrentfor every*.test.ts.The shared test harness gains an
isolateRepofixture (viait.scoped({ isolateRepo: true })in nesteddescribeblocks). When enabled,repois a freshly created repository withen-usas master and is deleted after the test; otherwise tests keep using the injected shared repo. Locale, push, status, and sync mutation tests are wrapped in this pattern and no longer callresetLocalesor other shared-repo teardown helpers.Test helpers
removeLocale,resetLocales, anddeleteAccessTokenare removed fromtest/prismic.tsas unused.locale-set-masterkeeps the “already master” case on the shared repo; only the mutating case uses isolation.init/repo-createtests takeonTestFinishedfrom the Vitest callback instead of a top-level import.Production change is limited to formatting of a
console.warninpushwhen screenshot deletion fails.Reviewed by Cursor Bugbot for commit 709ed5a. Bugbot is set up for automated code reviews on this repo. Configure here.
Footnotes
Please use these labels when submitting a review:
⚠️ #issue: Strongly suggest a change.
❓ #ask: Ask a question.
💡 #idea: Suggest an idea.
🎉 #nice: Share a compliment. ↩