Add pre-commit hook and CLAUDE.md - #3
Merged
Merged
Conversation
Husky + lint-staged runs prettier and eslint on staged files before every commit. CLAUDE.md documents the check workflow and project conventions.
4 tasks
4 tasks
jpr5
added a commit
that referenced
this pull request
Apr 22, 2026
## Summary Follow-up to #121 — the 19 migration-page quality items that were out-of-subject for the multi-turn docs + v1.14.4 release PR. All items per [this Notion page](https://www.notion.so/34a3aa3818528177bab7e549fa151a8f). **Stacked on #121** (base: \`docs/multi-turn-coverage\`). Once #121 merges, GitHub will retarget this to \`main\`. Highlights: - Fix \`npx @copilotkit/aimock -f …\` invocations across the migration guides (broken since the 1.7.0 bin split — route flag-driven usage through \`npx -p @copilotkit/aimock llmock …\`). - Replace Go / Rust SDK examples that used nonexistent builder APIs (\`sashabaranov/go-openai\` canonical \`DefaultConfig\` + \`NewClientWithConfig\`; \`async-openai\` canonical \`OpenAIConfig::new().with_api_base\`). - Add fail-fast to Kotlin + Python health-check loops that silently swallowed exceptions and let tests proceed against a dead server. - Harden python-mocks conftest fixtures: restore \`OPENAI_BASE_URL\` / \`OPENAI_API_KEY\` in finally; \`proc.wait(timeout=10)\` with kill fallback; \`DEVNULL\` instead of \`PIPE\` + no drain thread. - Correct invalid JSON in chaos-testing journal + piyook fixture examples. - Update stale MSW v2 WebSocket claim, add missing \`msw\` import. - Fix Python \`on_message\` kwarg mismatch (\`content=…\` → positional dict). - Remove unsupported Weaviate claim from VectorMock feature card. - Add \`livenessProbe\` / \`readinessProbe\` to helm \`values.yaml\` example. - Harden GitHub Actions docker teardown (\`--rm\` + \`if: always()\` + \`docker rm -f\`). - Wrap \`enableRecording\` programmatic example in try/finally with \`mock.stop()\` cleanup. Item #3 (docker-compose \`command: aimock --config …\` double-invocation) was refuted — no \`command: aimock\` patterns remain in the current tree; the Notion citation appears to have referenced a prior state. ## Test plan - [x] \`pnpm run format:check\` passes - [x] \`pnpm run lint\` passes - [x] \`pnpm run test\` — 2437 passed / 26 skipped - [ ] CI green
5 tasks
jpr5
added a commit
that referenced
this pull request
Apr 23, 2026
## Summary - Adds mokksy/ai-mocks to the COMPETITORS list in the updater script and surfaces a visible column in the front-page comparison table. - Links all 6 competitor headers in the comparison table to their GitHub repos (previously only mokksy had a link). - Corrects mokksy/ai-mocks feature cells based on their README: cross-process interception, chat completions SSE, responses API SSE, Claude Messages API, Gemini streaming, embeddings API, programmatic API (Kotlin/JVM), A2A agent mocking, multi-provider (5 providers), and dependencies (JVM). - Adds parseCurrentMatrix unit test asserting all 6 headers are extracted, correctly indexed, and that stripping anchor tags causes the parser to return 0 headers (regression guard). ## Why The competitive-matrix updater's header regex requires an <a> inside each <th>. With only mokksy linked, the parser saw 1 column instead of 6, producing zero-change dry-runs. Linking all headers fixes the parser and makes the table consistent. The mokksy cells were all initialized to "No" but the README clearly documents support for multiple features. ## Dry-run output Parser now correctly identifies 6 competitor columns and 32 capability rows. The 1 remaining auto-detected change (GitHub Action for mokksy) is a false positive from a CI badge URL in their README. ## Mokksy cells corrected (evidence from README) | Feature | Evidence | |---|---| | Cross-process interception | "mock HTTP server built with Kotlin and Ktor" -- real server | | Chat Completions SSE | "response streaming and Server-Side Events (SSE)", feature matrix shows Chat Completions YES for all providers | | Responses API SSE | Feature matrix shows "Responses" under OpenAI Additional APIs | | Claude Messages API | Anthropic listed as supported service #2 with Chat Completions and Streaming YES | | Gemini streaming | Google VertexAI Gemini listed as service #3 with Streaming YES | | Embeddings API | Feature matrix shows Embeddings YES for OpenAI and Ollama | | Programmatic API | Kotlin/JVM library with Maven Central artifacts | | A2A agent mocking | A2A Protocol listed as service #5 with full protocol support | | Multi-provider | 5 services: OpenAI, Anthropic, Gemini, Ollama, A2A | | Dependencies | JVM (Kotlin) -- not "No" | ## Test plan - [x] pnpm lint -- clean - [x] pnpm test -- 2492 passed, 36 skipped - [x] Dry-run: parser identifies 6 columns, 32 rows - [x] New parseCurrentMatrix tests: 4 assertions covering header count, index mapping, row data, and plain-text regression - [ ] CI green on this PR
2 tasks
jpr5
added a commit
that referenced
this pull request
Jun 24, 2026
… warn/opt-out (#276) ## TL;DR — what this is **What aimock does:** records real LLM API calls once, then replays the recorded responses in tests — no live API, deterministic, fast. **The change in one sentence:** matching a replay to a recorded fixture is now anchored on the **content** of the request (the messages/tools being sent), not on the **turn number** it happened to be recorded at. **Why:** the old logic treated "this is turn #3" as a hard requirement. If your test's conversation drifted by even one turn from when it was recorded, replay would **fail to match** and error out — brittle. Real conversations reorder, retry, and branch. Content is the stable identity; turn position is not. ### What actually changed - **Replay:** `turnIndex` is no longer a hard reject gate. A fixture matches if its **content** matches; turn position becomes a tiebreaker, not a veto. - **Record:** unchanged — still strict (records exactly what happened, in order). - **Visibility:** when a fixture matches at a *different* position than recorded, aimock emits a **divergence warning** (with a `matchedBy` reason), so drift is never silent. - **Escape hatch:** `AIMOCK_STRICT_TURN_INDEX=1` restores the old strict behavior. ### Risk surfaces (public pkg, ~500k downloads/wk) → mitigations | Risk | What could go wrong | Mitigation | | --- | --- | --- | | **Looser matching → wrong fixture** | Replay silently returns a fixture that isn't the right one | Content predicates are **primary** — a match still requires the request content to match. Position is only relaxed *after* content already matches. A truly wrong match needs two content-identical requests at different turns (rare outside showcase-style harnesses). | | **Silent false-green in tests** | A test passes against a mismatched fixture, hiding a real regression | The **divergence warn** makes every position-drift visible; `matchedBy` / `describeMatch` explain *why* a fixture matched. We never hide the drift — we surface it. | | **Breaking change on a minor bump** | Existing users depending on strict turn-index matching get different behavior | Change is **additive** (relaxation only as a fallback); the canonical case (record→replay at the same position) is byte-for-byte unchanged. `AIMOCK_STRICT_TURN_INDEX=1` opt-out preserves old behavior → ships as **minor, not major**. | | **Surprise / hard to debug** | "Why did this fixture match?" | New `matchedBy` field + a throttled, human-readable warning explain the match decision. | **Bottom line:** it trades a brittle, position-coupled gate for a content-anchored match that mirrors how conversations really behave — while keeping strict mode one env var away, recording untouched, and every divergence loud rather than silent. The empirical blast-radius check on our corpus showed **zero wrong-fixture matches**. --- ## Summary Makes aimock's fixture **replay** matching content-anchored: a fixture's recorded **content/shape** is the primary match key, and `turnIndex` is no longer a hard reject gate. This kills a class of false-misses where a multi-bubble / multi-turn run that drifts *past* its scripted `turnIndex` was spuriously rejected (`empty assistant response`), even though the content matched. The **record** path is unchanged — strict per-turn capture is preserved via `MatchOptions.strictTurnIndex`. ##⚠️ Public replay-semantics change — please confirm intended A fixture that previously **force-missed** on `turnIndex` will now **match** if its content predicates match. **Empirically measured** by replaying the real fixture corpora (showcase d6 + aimock examples) through the old vs new matcher — **786 fixture sets / 9769 requests**: - **3213 MISS→MATCH** (the intended false-miss fixes), **0 MATCH→MISS**, **0 wrong-fixture selections** (the 69 MATCH→DIFF are all same-content twins / forward-progress). - **0 changes at the canonical position** (`assistantCount == turnIndex`) — divergence happens *only* when a conversation runs off-by-N from its scripted turn. So this is a strict improvement on real corpora; it's a **minor** bump. (Content gating stays strict; multiple content matches → closest scripted `turnIndex`; all-ahead → unpositioned fallback.) ## Not surprising anyone (detection + reversible opt-out) For the divergence to be discoverable + reversible by the ~500k-downloads/wk user base: - **`AIMOCK_STRICT_TURN_INDEX=1`** — process opt-out that restores the legacy strict-turnIndex hard gate for replay (default = new content-anchored behavior). This reversible escape hatch is what keeps the change a minor bump. - **Surgical divergence warn** — a once-per-fixture, logger-gated `warn` fires only when a served fixture's `turnIndex` was relaxed (i.e. the strict gate would have rejected it). **Silent in programmatic use** (logger default), so it never spams a test suite; for the typical record→replay user (who sits at the canonical position) divergence is rare, so the warn is a genuine, useful signal. - **Diagnostic fields** `turnIndexRelaxed` / `matchedBy` on the match diagnostic for inspection. ## Review Converged via two full `cr-loop` passes (content-anchored matching, then the warn package) — 7 unbiased agents/round + confirmation rounds + Procedure-3 audits (0 promotions). Notable fixes caught in review: WeakSet-identity throttle (predicate/RegExp fixtures no longer collide), accurate `matchedBy` attribution, and the divergence-direction wording. ## Test plan - [x] `pnpm test` — **4041 passed / 0 failed**; `tsc --noEmit` clean; lint 0/0; build ok - [ ] **Confirm the intended replay semantics** (above) before merge ## Known follow-ups (pre-existing, not this PR) - `getSystemText`/`getTextContent` present-but-empty `""` stray-newline edge under `useExactMatch`. - Handler input-validation: unguarded `parts`/`content` derefs (gemini/cohere/responses/bedrock/bedrock-converse) → TypeError on malformed JSON. Not auto-merging — flagging the replay-semantics change for review.
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
## Summary - Adds mokksy/ai-mocks to the COMPETITORS list in the updater script and surfaces a visible column in the front-page comparison table. - Links all 6 competitor headers in the comparison table to their GitHub repos (previously only mokksy had a link). - Corrects mokksy/ai-mocks feature cells based on their README: cross-process interception, chat completions SSE, responses API SSE, Claude Messages API, Gemini streaming, embeddings API, programmatic API (Kotlin/JVM), A2A agent mocking, multi-provider (5 providers), and dependencies (JVM). - Adds parseCurrentMatrix unit test asserting all 6 headers are extracted, correctly indexed, and that stripping anchor tags causes the parser to return 0 headers (regression guard). ## Why The competitive-matrix updater's header regex requires an <a> inside each <th>. With only mokksy linked, the parser saw 1 column instead of 6, producing zero-change dry-runs. Linking all headers fixes the parser and makes the table consistent. The mokksy cells were all initialized to "No" but the README clearly documents support for multiple features. ## Dry-run output Parser now correctly identifies 6 competitor columns and 32 capability rows. The 1 remaining auto-detected change (GitHub Action for mokksy) is a false positive from a CI badge URL in their README. ## Mokksy cells corrected (evidence from README) | Feature | Evidence | |---|---| | Cross-process interception | "mock HTTP server built with Kotlin and Ktor" -- real server | | Chat Completions SSE | "response streaming and Server-Side Events (SSE)", feature matrix shows Chat Completions YES for all providers | | Responses API SSE | Feature matrix shows "Responses" under OpenAI Additional APIs | | Claude Messages API | Anthropic listed as supported service #2 with Chat Completions and Streaming YES | | Gemini streaming | Google VertexAI Gemini listed as service #3 with Streaming YES | | Embeddings API | Feature matrix shows Embeddings YES for OpenAI and Ollama | | Programmatic API | Kotlin/JVM library with Maven Central artifacts | | A2A agent mocking | A2A Protocol listed as service #5 with full protocol support | | Multi-provider | 5 services: OpenAI, Anthropic, Gemini, Ollama, A2A | | Dependencies | JVM (Kotlin) -- not "No" | ## Test plan - [x] pnpm lint -- clean - [x] pnpm test -- 2492 passed, 36 skipped - [x] Dry-run: parser identifies 6 columns, 32 rows - [x] New parseCurrentMatrix tests: 4 assertions covering header count, index mapping, row data, and plain-text regression - [ ] CI green on this PR
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
… warn/opt-out (#276) ## TL;DR — what this is **What aimock does:** records real LLM API calls once, then replays the recorded responses in tests — no live API, deterministic, fast. **The change in one sentence:** matching a replay to a recorded fixture is now anchored on the **content** of the request (the messages/tools being sent), not on the **turn number** it happened to be recorded at. **Why:** the old logic treated "this is turn #3" as a hard requirement. If your test's conversation drifted by even one turn from when it was recorded, replay would **fail to match** and error out — brittle. Real conversations reorder, retry, and branch. Content is the stable identity; turn position is not. ### What actually changed - **Replay:** `turnIndex` is no longer a hard reject gate. A fixture matches if its **content** matches; turn position becomes a tiebreaker, not a veto. - **Record:** unchanged — still strict (records exactly what happened, in order). - **Visibility:** when a fixture matches at a *different* position than recorded, aimock emits a **divergence warning** (with a `matchedBy` reason), so drift is never silent. - **Escape hatch:** `AIMOCK_STRICT_TURN_INDEX=1` restores the old strict behavior. ### Risk surfaces (public pkg, ~500k downloads/wk) → mitigations | Risk | What could go wrong | Mitigation | | --- | --- | --- | | **Looser matching → wrong fixture** | Replay silently returns a fixture that isn't the right one | Content predicates are **primary** — a match still requires the request content to match. Position is only relaxed *after* content already matches. A truly wrong match needs two content-identical requests at different turns (rare outside showcase-style harnesses). | | **Silent false-green in tests** | A test passes against a mismatched fixture, hiding a real regression | The **divergence warn** makes every position-drift visible; `matchedBy` / `describeMatch` explain *why* a fixture matched. We never hide the drift — we surface it. | | **Breaking change on a minor bump** | Existing users depending on strict turn-index matching get different behavior | Change is **additive** (relaxation only as a fallback); the canonical case (record→replay at the same position) is byte-for-byte unchanged. `AIMOCK_STRICT_TURN_INDEX=1` opt-out preserves old behavior → ships as **minor, not major**. | | **Surprise / hard to debug** | "Why did this fixture match?" | New `matchedBy` field + a throttled, human-readable warning explain the match decision. | **Bottom line:** it trades a brittle, position-coupled gate for a content-anchored match that mirrors how conversations really behave — while keeping strict mode one env var away, recording untouched, and every divergence loud rather than silent. The empirical blast-radius check on our corpus showed **zero wrong-fixture matches**. --- ## Summary Makes aimock's fixture **replay** matching content-anchored: a fixture's recorded **content/shape** is the primary match key, and `turnIndex` is no longer a hard reject gate. This kills a class of false-misses where a multi-bubble / multi-turn run that drifts *past* its scripted `turnIndex` was spuriously rejected (`empty assistant response`), even though the content matched. The **record** path is unchanged — strict per-turn capture is preserved via `MatchOptions.strictTurnIndex`. ##⚠️ Public replay-semantics change — please confirm intended A fixture that previously **force-missed** on `turnIndex` will now **match** if its content predicates match. **Empirically measured** by replaying the real fixture corpora (showcase d6 + aimock examples) through the old vs new matcher — **786 fixture sets / 9769 requests**: - **3213 MISS→MATCH** (the intended false-miss fixes), **0 MATCH→MISS**, **0 wrong-fixture selections** (the 69 MATCH→DIFF are all same-content twins / forward-progress). - **0 changes at the canonical position** (`assistantCount == turnIndex`) — divergence happens *only* when a conversation runs off-by-N from its scripted turn. So this is a strict improvement on real corpora; it's a **minor** bump. (Content gating stays strict; multiple content matches → closest scripted `turnIndex`; all-ahead → unpositioned fallback.) ## Not surprising anyone (detection + reversible opt-out) For the divergence to be discoverable + reversible by the ~500k-downloads/wk user base: - **`AIMOCK_STRICT_TURN_INDEX=1`** — process opt-out that restores the legacy strict-turnIndex hard gate for replay (default = new content-anchored behavior). This reversible escape hatch is what keeps the change a minor bump. - **Surgical divergence warn** — a once-per-fixture, logger-gated `warn` fires only when a served fixture's `turnIndex` was relaxed (i.e. the strict gate would have rejected it). **Silent in programmatic use** (logger default), so it never spams a test suite; for the typical record→replay user (who sits at the canonical position) divergence is rare, so the warn is a genuine, useful signal. - **Diagnostic fields** `turnIndexRelaxed` / `matchedBy` on the match diagnostic for inspection. ## Review Converged via two full `cr-loop` passes (content-anchored matching, then the warn package) — 7 unbiased agents/round + confirmation rounds + Procedure-3 audits (0 promotions). Notable fixes caught in review: WeakSet-identity throttle (predicate/RegExp fixtures no longer collide), accurate `matchedBy` attribution, and the divergence-direction wording. ## Test plan - [x] `pnpm test` — **4041 passed / 0 failed**; `tsc --noEmit` clean; lint 0/0; build ok - [ ] **Confirm the intended replay semantics** (above) before merge ## Known follow-ups (pre-existing, not this PR) - `getSystemText`/`getTextContent` present-but-empty `""` stray-newline edge under `useExactMatch`. - Handler input-validation: unguarded `parts`/`content` derefs (gemini/cohere/responses/bedrock/bedrock-converse) → TypeError on malformed JSON. Not auto-merging — flagging the replay-semantics change for review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
pnpm run format:checkpassespnpm run lintpassespnpm testpasses (259 tests)