Find what your AI agent gets wrong — before you have a rubric. Qualitative eval for PMs.
Binary safety verdicts (SAFE/HELD/LEAK/MISS/BROKE) + persona fan-out for LLM pipeline evals
Lightweight CLI for versioning prompts and running eval suites. Score outputs with deterministic matching or LLM-as-judge, compare prompt versions with rich terminal diffs. No infra, git-friendly, local-first.
Self-hosted evaluation framework for MCP servers. Define YAML test suites, run agents against MCP tools, score with LLM-as-judge rubrics, and monitor results in a FastAPI dashboard with tool-call traces and regression detection.
Web studio for evaluating LLM agents with rubric-based scoring, LLM-as-judge validation, and side-by-side run diffs. Trace agent execution, compare performance, and iterate with full visibility into tool calls and LLM interactions.
Self-hostable LLM evaluation framework for measuring model performance across configurable skills. Run YAML-defined benchmarks against OpenAI/Anthropic models, score with LLM-as-judge, compare results in a CLI and web dashboard.
A FastAPI WebSocket service and CI-friendly CLI for running LangSmith evaluations on demand — bundles LLM-as-judge (Claude) and heuristic evaluators, syncs datasets idempotently, and gates deploys via threshold checks.
Open-source evaluation framework for AI agents. Define test suites with rubrics, run your agent, get LLM-as-judge scores against criteria, inspect full execution traces, and diff runs to catch behavioral regressions.
Define YAML rubrics, run agents through test scenarios, get LLM-judged per-criterion scores with full trajectory traces, and analyze results in an interactive web dashboard.
Local CLI for evaluating and regression-testing LLM prompts. Define test suites in YAML with system prompts, expected outputs, and scoring rules. Run against Claude and GPT models to detect regressions and compare prompt quality across versions.
Agent evaluation framework: run LLM agents against datasets, capture execution traces, score with rubric-based LLM judges, and view regressions in a web dashboard. Local-first, no external infra required.
A small, reusable eval framework for scoring AI classifiers: blind classification via Claude Code, confusion matrices, ordinal miss-direction, confusable-pattern trap audits.
Add a description, image, and links to the eval-framework topic page so that developers can more easily learn about it.
To associate your repository with the eval-framework topic, visit your repo's landing page and select "manage topics."