Skip to content

test: run repository-mutating tests against per-test isolated repositories - #231

Merged
angeloashmore merged 3 commits into
mainfrom
aa/isolated-repo-tests
Jul 22, 2026
Merged

test: run repository-mutating tests against per-test isolated repositories#231
angeloashmore merged 3 commits into
mainfrom
aa/isolated-repo-tests

Conversation

@angeloashmore

@angeloashmore angeloashmore commented Jul 22, 2026

Copy link
Copy Markdown
Member

Resolves:

Description

Removes the serial test project. Tests that mutate whole-repository state (locales, push, sync, status) opt into a throwaway repository per test, so the entire suite runs concurrently:

it("supports --help", async ({ expect, prismic }) => {
	// Runs against the shared repository. No extra repository is created.
});

describe("with an isolated repository", () => {
	it.scoped({ isolateRepo: true });

	it("adds a locale", async ({ expect, prismic, repo }) => {
		// `repo` is a fresh repository unique to this test, deleted afterwards.
	});
});

Checklist

  • A comprehensive Linear ticket, providing sufficient context and details to facilitate the review of the PR, is linked to the PR.
  • If my changes require tests, I added them.
  • If my changes affect backward compatibility, it has been discussed.
  • If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

How to QA 1

Run node --run test. All 83 files (288 tests) run concurrently; there are no *.serial.test.ts files left.

🤖 Generated with Claude Code


Note

Low Risk
Changes are almost entirely test and Vitest configuration; runtime CLI behavior is unchanged aside from a whitespace-only edit in push.ts.

Overview
Replaces serial e2e with per-test isolated Prismic repositories so the whole suite can run concurrently. Vitest’s separate serial project and all *.serial.test.ts files are removed; a single config enables sequence.concurrent for every *.test.ts.

The shared test harness gains an isolateRepo fixture (via it.scoped({ isolateRepo: true }) in nested describe blocks). When enabled, repo is a freshly created repository with en-us as master and is deleted after the test; otherwise tests keep using the injected shared repo. Locale, push, status, and sync mutation tests are wrapped in this pattern and no longer call resetLocales or other shared-repo teardown helpers.

Test helpers removeLocale, resetLocales, and deleteAccessToken are removed from test/prismic.ts as unused. locale-set-master keeps the “already master” case on the shared repo; only the mutating case uses isolation. init / repo-create tests take onTestFinished from the Vitest callback instead of a top-level import.

Production change is limited to formatting of a console.warn in push when screenshot deletion fails.

Reviewed by Cursor Bugbot for commit 709ed5a. Bugbot is set up for automated code reviews on this repo. Configure here.

Footnotes

  1. Please use these labels when submitting a review:
    ❓ #ask: Ask a question.
    💡 #idea: Suggest an idea.
    ⚠️ #issue: Strongly suggest a change.
    🎉 #nice: Share a compliment.

angeloashmore and others added 2 commits July 22, 2026 00:53
…ories

Tests that mutate whole-repository state (locales, push, sync, status)
previously ran serially against the shared repository. They now create a
throwaway repository per test via an it.scoped repo override, letting the
whole suite run concurrently. Tests that do not touch repository state,
like --help and error-path tests, keep the shared repository.

The isolatedRepo fixture takes no other fixtures as dependencies because
vitest resolves it.scoped override dependencies only against other scoped
fixtures, silently passing undefined for anything from test.extend.

Also flattens the vitest config to a single test config now that the
serial project is gone. This enabled the previously inert root-level
typecheck option and surfaced that concurrent tests must use
onTestFinished from the test context rather than the global import.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n fixture

Suites opt into a throwaway repository with it.scoped({ isolateRepo: true })
instead of overriding repo with a standalone function. The repo fixture
branches on the flag and uses normal fixture dependencies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@angeloashmore
angeloashmore marked this pull request as ready for review July 22, 2026 01:49

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9fbb2a0. Configure here.

Comment thread test/it.ts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@lihbr lihbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice to me :)

@angeloashmore
angeloashmore merged commit c25b13b into main Jul 22, 2026
35 of 42 checks passed
@angeloashmore
angeloashmore deleted the aa/isolated-repo-tests branch July 22, 2026 02:15
angeloashmore added a commit that referenced this pull request Jul 22, 2026
Adopts main's isolateRepo fixture API from #231: drops the serial test
project split, moves the exec fixture into eval/it.ts, and switches the
eval repo isolation to isolateRepo: true.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
angeloashmore added a commit that referenced this pull request Jul 22, 2026
Evals only shell out to git; the generic runner's options plumbing was
left over from the pre-#231 CLI fixture. throwOnError surfaces setup
failures that previously passed silently.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
angeloashmore added a commit that referenced this pull request Jul 28, 2026
* feat: add AI eval harness for the CLI

Plain vitest evals that drive a real Claude Code agent (via the Agent SDK)
against the built CLI in a generic, hermetic config. Adds the `agent` fixture,
an `toHaveRun` matcher, an LLM-judge matcher with a never-regress ratchet, and a
per-run reporter that records efficiency metrics and the agent's prismic calls.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(eval): consolidate harness into one fixture file

Merge the agent fixture, matchers, judge, config isolation, and result
recording into eval/it.ts; delete claude.ts, report.ts, and setup.ts.

Move the judge ratchet from a read-back of results.jsonl into an inline
threshold literal that vitest -u rewrites like a snapshot, so results.jsonl
is write-only. Trim recorded fields to effectiveness and effort (turns,
tokens, duration); drop cost, which swings with provider pricing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(eval): drop explanatory comments from harness

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(eval): harden judge, run isolation, and bar ratchet

- Judge uses the SDK's native structured output instead of hand-parsing JSON;
  score is range-checked to [0, 1]
- Create a throwaway Claude config dir per agent/judge call and clean it up;
  set persistSession: false so runs stay ephemeral
- Detect -u from vitest's update state; the worker's process.argv is empty
- Fix raiseLiteral rewriting the wrong literal on the callsite line, and
  support inline bars
- Fix the evals npm script so vitest also receives RUN_EVALS

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(eval): score by pass rate over trials with a binary judge

Each eval now runs EVAL_TRIALS times (default 3) via it.for; a custom
reporter prints per-eval pass rates with infra failures excluded and
appends per-trial rows (pass, cost, tokens, turns, commands) to
results.jsonl. The judge is binary like agent-eval's: prose criteria,
pass/fail verdicts, no scores, no ratcheting bars. The full eval catalog
is implemented as it.skip drafts; only the rich-text smoke eval runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(eval): add results digest script

`node --run evals:report` prints one block per eval with one line per
run (pass rate, cost, mean tokens/turns/duration) and a latest-run
footer with best-of and per-trial rates. Rows now carry a per-run id so
the digest groups runs exactly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(eval): tag results with CLI commit and cap history

Rows record the git SHA that produced them, shown in the digest so rate
changes point at a commit range. The file keeps the newest 100 runs; the
digest shows the last 10 per eval. results.jsonl merges with the union
driver so branch runs never conflict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): simplify harness

The judge now calls the Messages API directly with structured outputs
instead of spawning a Claude Code session. The agent fixture writes its
own task.meta via the built-in task context, trial rows no longer carry
a parsed trial number, and eval env vars are parsed with a zod schema.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(eval): validate evals, isolate repos, and run concurrently

Enable 26 validated evals with per-trial runs and skip 3 that fail
consistently, with the reason noted on each. Trim duplicate evals and
merge the two slice component evals into one.

Each eval now gets its own repository via an isolatedRepo fixture,
letting evals run 8-way concurrent without cross-test interference.
Global teardown sweeps orphaned repositories created by agents
(e.g. prismic init) that the harness cannot name in advance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(eval): add happy-path evals and count only real command runs

Add four evals from the Next.js happy-path docs: init with an existing
repository, page types vs custom types, commit-then-push flow, and
production previews after a deploy. The last two fail today and stay
enabled: evals are measurements, and their pass rates track CLI gaps.
Un-skip the other agent-capability evals for the same reason; only the
field rename eval stays skipped since the CLI cannot rename fields.

toHaveRun now evaluates each shell segment separately and ignores
--help invocations, so reading help no longer counts as running a
command in either direction.

Scope the results digest to the last EVAL_HISTORY runs (default 10)
with a per-run footer, and document the results.jsonl schema.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): replace exec fixture with a git-only fixture

Evals only shell out to git; the generic runner's options plumbing was
left over from the pre-#231 CLI fixture. throwOnError surfaces setup
failures that previously passed silently.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style(eval): drop explanatory comments from harness

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): split evals into behavior-named files under evals/

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(test): gate evals with an explicit opt-in and drop test-mode builds

Evals now require PRISMIC_ALLOW_EVALS=true and an explicit --project evals;
bare vitest runs only unit tests. Tests run against the prebuilt dist (no
MODE=test build), sync's poll interval is a validated PRISMIC_SYNC_POLL_MS
env var, agent-created repos are cleaned up per test instead of an
account-wide sweep, and dead MODE/DEV env plumbing is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): correct results.jsonl gitattributes path and drop unused helper

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: drop redundant .env.test.local gitignore entry

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): use toLocaleString for run dates and trim mean guard

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): type results.jsonl rows at the write site

Export AgentRecord with field docs, build Row from it in the reporter,
and reuse that type in the report instead of a duplicated schema comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): define results.jsonl rows as a zod schema and parse on read

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style(eval): use TSDoc comments on the row schema

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(eval): judge field names in design-sensible-models bars

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(eval): model a slice from a screenshot

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(eval): record eval run at 6c4397b

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): apply sequence.concurrent at the config root

Vitest silently ignores `sequence` inside a `projects` entry, so every
test in a file ran serially. Root-level `sequence.concurrent` propagates
to projects and restores within-file concurrency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(eval): source agent guidance from the pinned prismicio/skills SKILL.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): simplify the harness to two files and a per-run results.json

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: drop the union-merge rule for the removed results.jsonl

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(eval): record filtered runs to results.local.json only

Partial runs (file filters, -t, it.only) are detected by the reporter and
never overwrite results.json, so it always holds one full run. Permanent
disables must be it.todo, which vitest keeps distinguishable from filter
skips. Also fixes the remaining bin/ mkdir race in the project fixture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(eval): write results.json on a single line

Every run rewrites most values, so a pretty-printed file only inflates PR
diffs; one line keeps run-over-run changes to one changed line. Read it
with jq or node --run evals:report.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): use object-form fixtures and revert to vitest 4.0.17

Aligns the eval fixtures with test/it.ts and drops the vitest 4.1
builder-pattern API, shrinking the lockfile diff to the agent SDK.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(eval): drop the .ts import extension and restore main's tsconfig

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(eval): skip Bash tool calls without a command string

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): stop forcing throwOnError in the exec fixture

Tests that assert non-zero exit codes need the result object back, not a
thrown error. Restores main's behavior; the eval git setup surfaces
failures through its own assertions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style(test): drop the sequence placement comment

* fix(eval): keep results.json untouched when a trial never reached the agent

A fixture failure before the prompt leaves a failed test with no recorded
trial. Recording the rest as a full run would misstate pass rates, so such
runs now count as partial.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: fetch the profile for telemetry only when telemetry is enabled

Every CLI invocation fired a fire-and-forget /profile request even with
telemetry disabled, since the call sat outside the telemetry gates. In CI
this put every spawned CLI process on user-service concurrently, which
starts returning 500s at roughly eight concurrent same-user requests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style(eval): drop the stale Fails comments

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: run write-token tests sequentially

Wroom's settings/security/token endpoint intermittently returns 500 when
the same user fires concurrent write-token creates, which the concurrent
test mode did across CI matrix jobs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants