Skip to content

Add Sabot to LLM & MCP Testing#101

Open
Jott2121 wants to merge 1 commit into
cleder:mainfrom
Jott2121:add-sabot
Open

Add Sabot to LLM & MCP Testing#101
Jott2121 wants to merge 1 commit into
cleder:mainfrom
Jott2121:add-sabot

Conversation

@Jott2121

@Jott2121 Jott2121 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Adding Sabot under LLM & MCP Testing, placed alphabetically between mcp-server-fuzzer and Tenro.

What it does that the neighbouring entries don't: Tenro and mcp-server-fuzzer exercise agents from the outside. Sabot injects one controlled fault inside a running multi-agent pipeline (a corrupted tool result, a falsified success report, an altered inter-agent message, a silent model downgrade, stale context, a silent no-op) and scores whether the pipeline's own reviewer and guardrail stages catch it — no external monitor in the detection path.

Why it may be useful to this list's readers: measured across LangGraph, CrewAI and AutoGen/Magentic-One, the median own-check detection rate was 16.7%, and the most common outcome was a correct final answer with nothing ever flagging the fault.

  • Apache-2.0 harness, CC BY 4.0 spec and results
  • Specification, seeds and adjudication anchors registered publicly before any scored run
  • Full raw trace corpus published, so every score is recomputable offline with stdlib Python and no API key: python3 scripts/score_wave2.py --seeds-file ../seeds/wave2.json reproduces all 189 published rows byte-identically
  • Archived with a DOI: https://doi.org/10.5281/zenodo.21539796

Disclosure: I'm the author. You merged my other project (Crucible) into Mutation Testing a couple weeks ago, happy to adjust the wording or drop this if it's out of scope for the section.

Sabot injects one controlled fault into a running multi-agent pipeline and scores
whether that pipeline's own reviewer and guardrail stages detect it, rather than
whether an external harness can. Apache-2.0 harness, pre-registered specification,
and the full raw trace corpus are published so the scores are recomputable offline.
@semanticdiff-com

Copy link
Copy Markdown

Review changes with  SemanticDiff

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@mergify

mergify Bot commented Jul 24, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@llamapreview llamapreview Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LlamaPReview — No blocking issues found

Review details and evidence

LlamaPReview checks

  • Reviewed changed regions in README.md.
  • Read the complete PR-head file README.md.

Automated review by LlamaPReview · Free for public open-source projects.

@Jott2121

Jott2121 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Note on the two red checks — I looked into both and neither is caused by this change, which is a single added line:

rumdl, typos, md-dead-link-check (the Action) and the whitespace/EOF hooks all passed. Happy to rebase if that would help clear them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant