Skip to content

feat: guard generic edit batch commit boundaries - #250

Merged
OBenner merged 60 commits into
developfrom
codex/generic-edit-batch-boundary-guards
May 18, 2026
Merged

feat: guard generic edit batch commit boundaries#250
OBenner merged 60 commits into
developfrom
codex/generic-edit-batch-boundary-guards

Conversation

@OBenner

@OBenner OBenner commented May 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add provider-specific negative fixtures for OpenAI, Gemini/Google, OpenRouter, LiteLLM, ZhipuAI, and Ollama so provider e2e diagnostics no longer rely only on generic unsupported-tool probes.
  • Add runtime policy and eval matrices to runtime modes output so planner/coder/QA runtime selection, fallback rules, required full-runtime paths, and eval artifacts are explicit.
  • Persist provider e2e smoke history in .auto-Codex/provider-smoke-history.json, recover corrupt history artifacts, and surface the run-history evidence in CLI/UI diagnostics.
  • Add runtime_eval_history to the runtime diagnostics control plane so persisted provider smoke evidence is summarized alongside the policy/eval matrix.
  • Add cli_runner_contract_matrix so every full-runtime CLI candidate is measured against the same run, cancel, resume, artifacts, event parser, and cost/account contract.
  • Add runtime_subagent_mutation_policy so mutating child agents are explicitly blocked until transaction boundaries, conflict-aware merge, and parent-approved apply/abort exist.
  • Add mcp_bridge_permission_matrix so local, external, and custom MCP bridge permissions expose policy metadata, audit artifacts, mutating permissions, strict allowlist status, and missing gates.
  • Normalize MCP bridge tool results into normalized_result with text, typed content, structured payloads, and error state for observations/audit/recovery consumers.
  • Classify external MCP tools/list lifecycle failures with failure_stage and failure_kind so UI/contract smoke can distinguish invalid responses, startup failures, timeouts, and server errors.
  • Classify bridged MCP tools/call failures in runtime observations and audit artifacts with the same lifecycle taxonomy.
  • Add a provider-neutral generic CLI runtime core for planned CLI runners with configurable run/cancel, JSONL event parsing, sanitized account summaries, and runner-specific artifacts.
  • Add a consolidated runtime_capability_matrix and Settings control-plane row for provider readiness, blockers, warnings, and CLI candidates.
  • Add runtime_comparative_eval_matrix and Settings comparative eval rows for Claude/Codex/OpenAI/Gemini/Ollama quality/cost/safety evidence.
  • Add a Generic Edit staged commit guard so commit_batch blocks if active staged postimages drift or become unverifiable before the batch closes, while rollback/abort mark restored snapshots as no longer staged.
  • Preserve staged commit guard status and drift paths in Generic Edit events and manifest recovery timelines for UI repair/rollback flows.
  • Block Generic Edit resume preflight on incomplete referenced mutation snapshots, not just missing/corrupt snapshot artifacts.
  • Add staged guard status and drift-path aggregation to provider smoke transaction batch diagnostics.
  • Surface staged guard statuses and drift paths in Settings transaction batch diagnostics.
  • Add explicit Generic Edit batch lifecycle events for begin/commit/abort/blocked transitions in result and manifest batch summaries.
  • Aggregate batch lifecycle actions/statuses in provider smoke diagnostics and surface them in Settings transaction batch rows.
  • Block Generic Edit resume preflight when manifest batch lifecycle counts drift from trace-derived batch lifecycle state.
  • Mark staged mutation snapshots as committed on successful commit_batch, preserving rolled_back state for abort/rollback snapshots.
  • Add batch_committed timeline events with committed snapshot ids to Generic Edit event and manifest artifacts.
  • Surface committed mutation snapshot ids in provider smoke transaction batch diagnostics and Settings rows.
  • Block Generic Edit resume when checkpoint or session-state resume inputs point at non-canonical artifact paths.
  • Carry committed mutation snapshot ids and commit operation ids through transaction summaries, batch summaries, manifests, and runtime events.
  • Surface Generic Edit batch commit operation ids in provider smoke diagnostics and Settings transaction-batch rows.
  • Block Generic Edit preflight/resume when manifest event_count drifts from the persisted event artifact.
  • Add a provider e2e transaction_batch_probe and transaction_batches reliability case for begin/commit batch coverage.
  • Add true Generic Edit isolated staged workspace handling: open batches temporarily materialize staged postimages for reads/next mutations, restore the real workspace baseline after each action, and apply staged postimages only on commit_batch.
  • Surface staged isolation status, workspace restore status, and baseline paths through provider smoke diagnostics and Settings transaction-batch rows.
  • Preserve staged workspace materialized/restored/batch-id events in Generic Edit manifests and surface staged mutation/path/lifecycle counts in the artifact viewer.
  • Surface Generic Edit batch lifecycle events, boundary blockers, required next actions, and resolution strategies in the artifact viewer.
  • Report corrupt/invalid Generic Edit mutation snapshot artifacts with structured corrupt_json / invalid_schema preflight health reasons.
  • Silence the Sonar security false positive on a pytest tmp_path manifest rewrite with a scoped NOSONAR comment.
  • Split Generic Edit artifact-manifest transaction-batch normalization helpers to reduce Sonar cognitive complexity while preserving the IPC contract.
  • Block Generic Edit resume preflight when an isolated staged snapshot did not restore the workspace baseline or lost its staged baseline preimages.
  • Surface provider negative fixtures, provider history, runtime policy/eval rows, runtime eval history, CLI runner contract status, MCP permission gates, and mutating-subagent gates in the Settings runtime diagnostics control plane.
  • Keep the provider runtime roadmap current in docs and in this PR table.

Remaining Full-Autonomous Provider Plan

Area Status after this PR Remaining
Runtime foundation Done Keep compatibility metadata current as more CLI runners land.
Generic Edit v2 core Strong partial: transaction groups, batches, explicit batch lifecycle events, isolated staged workspace materialization/restoration, commit-time staged postimage apply, committed/rolled-back staged snapshot state, committed snapshot timeline/diagnostic events, commit operation ids in transaction/batch summaries, staged metadata, recovery policy, boundary guards, pre-execution staged isolation guards, pre-commit staged baseline drift guards, staged guard/drift-path timeline events, staged workspace materialized/restored/batch-id manifest events, rollback/repair with restored snapshot state, resume checkpoints, canonical resume-input guards, drift/corrupt/incomplete-artifact preflight, corrupt/invalid mutation snapshot health reasons, isolated staged snapshot integrity blockers, lifecycle-count and event-count manifest validation, manifest consistency, rich events, provider diagnostics, and Settings lifecycle rows are wired. Stronger UI-driven repair/rollback workflows and broader non-happy-path staged-overlay recovery hardening.
Generic autonomous API runtime Partial: generic_edit has JSON/native tool loops, local actions, MCP bridge calls, read-only subagents, fallback, smoke diagnostics, transaction batch contract diagnostics, recovery/batch guards, and provider-specific negative fixture probes. Live provider e2e against real OpenAI/Gemini/OpenRouter/LiteLLM/ZhipuAI/Ollama accounts for native tool calls/results, unsupported tools, gateway/model limitations, and recovery loops.
MCP Bridge v1 Strong partial: bridge plans, external health, Context7 execution, Graphiti/Linear/Electron/Puppeteer/custom adapter readiness, unavailable-tool observations, audit metadata, mcp_bridge_permission_matrix, normalized MCP tool results, and classified tools/list plus bridged tools/call lifecycle failures are wired. The runtime enforces RuntimeMcpToolPolicy before bridged tool execution and reports strict allowlist status. Broader live execution coverage beyond configured adapters, continuous arbitrary schema normalization, safer lifecycle/session reuse, and per-server execution smoke.
Provider reliability Strong partial: provider smoke now reports tool-loop contract, transaction batch contract, batch lifecycle actions/statuses, staged isolation/restore evidence, committed mutation snapshot ids, commit operation ids, boundary guards, staged drift guard details, resume policy, gateway/model limits, fallback, unsupported tools, open batches, provider-specific negative fixtures, persisted provider run history, corrupt-history repair, provider e2e transaction batch probe, and UI rows for all of those signals including staged drift paths. Optional live fault-injection suites, per-provider evidence from real external gateways, and richer trend views from accumulated history.
Subagent Orchestrator v2 Partial: read-only child sessions have bounded orchestration and runtime_subagent_mutation_policy now exposes the missing gates for mutating children. Implement transactional boundaries for mutating child contexts, conflict-aware merge, and parent-approved apply/abort so the policy can move from blocked to enabled.
CLI full-runtime class Strong partial: Codex CLI is wired and cli_runner_contract_matrix now tracks run/cancel/resume/artifacts/event_parser/cost_account readiness for every candidate. Planned runners now share a generic CLI core for configurable process execution, cancel, JSONL event parsing, sanitized account summaries, and <runner>_result/timeline/stdout/events artifacts. Add runner-specific command builders, resume semantics, and live smoke/e2e coverage for Aider/OpenCode/Goose/Gemini CLI/Qwen Code so they move from generic-core partial to ready.
Frontend control plane Strong partial: Settings diagnostics now shows tool-loop contract, transaction batch contract, batch lifecycle rows, committed snapshot and commit-operation rows, staged guard/drift-path rows, staged isolation/restore/baseline rows, staged workspace materialized/restored timeline events, Generic Edit artifact-viewer staged mutation/path/lifecycle rows, batch lifecycle events, boundary blockers, required next actions, resolution strategies, resume gates, provider health, MCP bridge status, MCP permission gates, subagent/runtime matrices, smoke controls, provider e2e transaction-batch probe labels, provider negative fixtures, provider run history, runtime policy/eval rows, consolidated runtime capability readiness/blockers/warnings, runtime eval history, CLI runner contract status, and mutating-subagent gates. Cost estimates, deeper provider controls, trend views, and deeper artifact drilldowns.
Policy/evals Strong partial: runtime recommendations, compatibility diagnostics, runtime_policy_matrix, runtime_eval_matrix, runtime_eval_history, runtime_comparative_eval_matrix, and UI control-plane rows are wired. Real cost/quality/safety metrics from live eval runs instead of mostly presence/status evidence, plus richer comparative reporting over time.

Validation

  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "manifest_event_preserves_staged_workspace_fields"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "batch or resume_preflight or mutation_snapshot or partial_failure or rollback or manifest_event_preserves_staged_workspace_fields"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "corrupt_mutation_snapshots or invalid_mutation_snapshot_schema"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "resume_preflight or corrupt_mutation_snapshots or invalid_mutation_snapshot_schema or mutation_snapshot or batch or partial_failure or rollback"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "manifest_batch_lifecycle_mismatch or corrupt_mutation_snapshots or invalid_mutation_snapshot_schema"
  • npm --prefix apps/frontend test -- spec-file-readers.test.ts GenericEditArtifactsPanel.test.tsx
  • npm --prefix apps/frontend run typecheck
  • cd apps/frontend && npm exec -- biome check src/main/ipc-handlers/task/spec-file-readers.ts src/main/ipc-handlers/task/spec-file-readers.test.ts src/renderer/components/task-detail/GenericEditArtifactsPanel.tsx src/renderer/components/task-detail/__tests__/GenericEditArtifactsPanel.test.tsx src/shared/types/task.ts src/shared/i18n/locales/en/tasks.json src/shared/i18n/locales/fr/tasks.json
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m ruff check apps/backend/agents/runtime/adapters/generic_edit.py tests/test_agent_runtime.py
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m ruff format apps/backend/agents/runtime/adapters/generic_edit.py tests/test_agent_runtime.py --check --diff
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m py_compile apps/backend/agents/runtime/adapters/generic_edit.py
  • node -e "JSON.parse(require('fs').readFileSync('apps/frontend/src/shared/i18n/locales/en/tasks.json','utf8')); JSON.parse(require('fs').readFileSync('apps/frontend/src/shared/i18n/locales/fr/tasks.json','utf8'))"
  • git diff --check
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_provider_smoke_command.py tests/test_runtime_modes_command.py -q
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "batch or resume_preflight or mutation_snapshot or partial_failure or rollback"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "unrestored_isolated_staged_snapshot or incomplete_mutation_snapshot"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "manifest_batch_lifecycle_mismatch"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "records_committed_batch_protocol or aborts_open_batch_with_snapshot_rollback"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_provider_smoke_command.py tests/test_agent_runtime.py -q -k "committed_batch_snapshots or records_committed_batch_protocol or batch or resume_preflight or mutation_snapshot or partial_failure or rollback"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_provider_smoke_command.py -q -k committed_batch_snapshots
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_provider_smoke_command.py -q
  • npm --prefix apps/frontend test -- provider-smoke-diagnostics.test.ts ProviderSettingsSection.test.tsx
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "records_committed_batch_protocol or batch or resume_preflight or mutation_snapshot or partial_failure or rollback"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "resume_preflight or trace_input_mismatch or rejects_resume_when_trace_mismatches_checkpoint or resumes_from_recovery_checkpoint or resumes_from_session_state_manifest"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "manifest_event_count_mismatch or resume_preflight or trace_input_mismatch or manifest_batch or batch or mutation_snapshot or partial_failure or rollback"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "batch or resume_preflight or mutation_snapshot or partial_failure or rollback"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_provider_smoke_command.py -q
  • npm --prefix apps/frontend test -- provider-smoke-diagnostics.test.ts ProviderSettingsSection.test.tsx
  • npm --prefix apps/frontend run typecheck
  • cd apps/frontend && npm exec -- biome check src/main/ipc-handlers/provider-smoke-diagnostics.ts src/main/ipc-handlers/provider-smoke-diagnostics.test.ts src/renderer/components/settings/ProviderSettingsSection.tsx src/renderer/components/settings/ProviderSettingsSection.test.tsx src/shared/types/settings.ts src/shared/i18n/locales/en/settings.json src/shared/i18n/locales/fr/settings.json
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_provider_smoke_command.py tests/test_agent_runtime.py -q -k "staged_batch_drift or transaction_batches_expose_staged_mutation_metadata or batch or resume_preflight or mutation_snapshot or partial_failure or rollback"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "mcp_bridge or external_mcp or normalize_mcp_tool_result"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_agent_runtime.py -q -k "codex_cli or cli_runtime_process or generic_cli or runtime_factory_can_create_generic_cli_runner"
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_runtime_modes_command.py -q
  • npm --prefix apps/frontend test -- ProviderSettingsSection.test.tsx
  • npm --prefix apps/frontend test -- provider-smoke-diagnostics.test.ts ProviderSettingsSection.test.tsx provider-smoke-runtime.test.ts
  • npm --prefix apps/frontend run typecheck
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/pytest tests/test_provider_smoke_command.py -q -k staged_batch_drift
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m ruff check apps/backend/cli/provider_smoke_commands.py apps/backend/cli/runtime_commands.py tests/test_provider_smoke_command.py tests/test_runtime_modes_command.py
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m ruff check apps/backend/agents/runtime/adapters/generic_edit.py apps/backend/cli/provider_smoke_commands.py tests/test_agent_runtime.py tests/test_provider_smoke_command.py
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m ruff check apps/backend/agents/runtime/__init__.py apps/backend/agents/runtime/adapters/__init__.py apps/backend/agents/runtime/adapters/generic_cli.py apps/backend/cli/runtime_commands.py tests/test_agent_runtime.py tests/test_runtime_modes_command.py
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m ruff format apps/backend/cli/provider_smoke_commands.py apps/backend/cli/runtime_commands.py tests/test_provider_smoke_command.py tests/test_runtime_modes_command.py --check --diff
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m ruff format apps/backend/agents/runtime/adapters/generic_edit.py apps/backend/cli/provider_smoke_commands.py tests/test_agent_runtime.py tests/test_provider_smoke_command.py --check --diff
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m ruff format apps/backend/agents/runtime/__init__.py apps/backend/agents/runtime/adapters/__init__.py apps/backend/agents/runtime/adapters/generic_cli.py apps/backend/cli/runtime_commands.py tests/test_agent_runtime.py tests/test_runtime_modes_command.py --check --diff
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m py_compile apps/backend/agents/runtime/__init__.py apps/backend/agents/runtime/adapters/__init__.py apps/backend/agents/runtime/adapters/generic_cli.py apps/backend/cli/runtime_commands.py
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m py_compile apps/backend/cli/provider_smoke_commands.py apps/backend/cli/runtime_commands.py
  • /Users/om/PycharmProjects/Auto-Coding/.venv/bin/python -m py_compile apps/backend/agents/runtime/adapters/generic_edit.py apps/backend/cli/provider_smoke_commands.py
  • node -e "JSON.parse(require('fs').readFileSync('apps/frontend/src/shared/i18n/locales/en/settings.json','utf8')); JSON.parse(require('fs').readFileSync('apps/frontend/src/shared/i18n/locales/fr/settings.json','utf8')); console.log('settings locale JSON ok')"
  • cd apps/frontend && npm exec -- biome check src/main/ipc-handlers/provider-smoke-diagnostics.ts src/main/ipc-handlers/provider-smoke-diagnostics.test.ts src/main/ipc-handlers/settings-handlers.ts src/renderer/components/settings/ProviderSettingsSection.tsx src/renderer/components/settings/ProviderSettingsSection.test.tsx src/shared/types/settings.ts src/shared/i18n/locales/en/settings.json src/shared/i18n/locales/fr/settings.json
  • cd apps/frontend && npm exec -- biome check src/renderer/components/settings/ProviderSettingsSection.tsx src/renderer/components/settings/ProviderSettingsSection.test.tsx src/shared/i18n/locales/en/settings.json src/shared/i18n/locales/fr/settings.json
  • git diff --check

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant