Add Sabot to LLM & MCP Testing#101
Conversation
Sabot injects one controlled fault into a running multi-agent pipeline and scores whether that pipeline's own reviewer and guardrail stages detect it, rather than whether an external harness can. Apache-2.0 harness, pre-registered specification, and the full raw trace corpus are published so the scores are recomputable offline.
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Tick the box to add this pull request to the merge queue (same as
|
There was a problem hiding this comment.
LlamaPReview — No blocking issues found
Review details and evidence
LlamaPReview checks
- Reviewed changed regions in
README.md. - Read the complete PR-head file
README.md.
Automated review by LlamaPReview · Free for public open-source projects.
|
Note on the two red checks — I looked into both and neither is caused by this change, which is a single added line:
|
Adding Sabot under LLM & MCP Testing, placed alphabetically between
mcp-server-fuzzerandTenro.What it does that the neighbouring entries don't: Tenro and mcp-server-fuzzer exercise agents from the outside. Sabot injects one controlled fault inside a running multi-agent pipeline (a corrupted tool result, a falsified success report, an altered inter-agent message, a silent model downgrade, stale context, a silent no-op) and scores whether the pipeline's own reviewer and guardrail stages catch it — no external monitor in the detection path.
Why it may be useful to this list's readers: measured across LangGraph, CrewAI and AutoGen/Magentic-One, the median own-check detection rate was 16.7%, and the most common outcome was a correct final answer with nothing ever flagging the fault.
python3 scripts/score_wave2.py --seeds-file ../seeds/wave2.jsonreproduces all 189 published rows byte-identicallyDisclosure: I'm the author. You merged my other project (Crucible) into Mutation Testing a couple weeks ago, happy to adjust the wording or drop this if it's out of scope for the section.