A behavioural test runner for Agent Skills. It answers the question your linter cannot: does this skill activate at the right time, call the right tools, produce the right artifacts — and still do all of that after you change the model or edit one paragraph of the prompt?
npx skillproof test ./skills
Skills are prompts in a trench coat. They get copied out of a registry into a repo, wired into an agent that can read files and run shell commands, and then never touched again. Nobody writes a test, because until now there was nothing to write a test with.
The scale of this is measured. From Registry to Repository: A Large-Scale Study of Agent Skills (arXiv:2607.00911) surveyed 18,463 registry skills and 23,199 personal skills, and reported:
| Reported figure | Value |
|---|---|
| Skills never updated after being copied | 53% |
Skills that declare allowed-tools |
~12–15% |
| Skills that declare a license | ~9–16% |
(Those are the paper's numbers, not ours — SkillProof does not survey registries. We cite them because they describe the failure mode this tool exists to fix.)
The consequences are boring and expensive: a skill that silently stops activating after a
model upgrade; a skill that quietly reaches for Bash on a request that should have been
read-only; a skill whose output shape drifts and breaks the three scripts downstream of it.
SkillProof turns a skill into something you can assert on, in CI, offline, on every PR.
npx skillproof init ./skills/pdf-extract # scaffold skillproof.yaml + fixtures + mock agent
npx skillproof test ./skills # run every manifest it can findinit writes a manifest next to your SKILL.md, a fixtures/basic/ directory, and a
mock.agent.json so the first run is green, offline, and takes under a second. Then you
start deleting the mock's assumptions until the tests describe what you actually want.
Prefer a local checkout:
npm install --save-dev skillproof
npx skillproof test ./skills --agents mock --mode liveskillproof 10 skills · 24 cases · agents: claude-code, mock
pdf-extract
✓ activates on a pdf extraction request claude-code 1.2s 1.2k tok
✗ blocks forbidden shell command claude-code 0.9s
tools.forbidden: expected no call to `Bash`, got Bash("rm -rf /")
SkillProof: 18/20 passed
✓ Activation precision: 96%
✓ Artifact contract: passed
✗ Safety: attempted forbidden shell command
⚠ Token cost increased 42% vs main
The four summary lines are generated by one shared function, so the terminal, the markdown report, the PR comment and the job summary are always byte-identical.
The Artifact contract and Safety lines report ⚠ … not verified rather than ✓ passed
whenever the checks behind them did not actually run to a green finish — a case that
errored, one --bail cut short, or a check that had to be skipped. A tools.forbidden
assertion passes trivially when the agent crashed before calling any tool, and a green
safety headline on a broken run is the last thing this tool should print.
Every key lives under a case's expect:. Full semantics in
docs/manifest-reference.md.
| Key | Type | Asserts |
|---|---|---|
activation |
required | forbidden | any |
The skill did / did not load for this prompt (name-normalised, case-insensitive). This is the one most skills fail. |
tools.allowed |
string[] |
Every tool call is in this set. First violation is reported with a 200-char input preview. |
tools.required |
string[] |
Each named tool was called at least once. |
tools.forbidden |
string[] |
None of these were called. Powers the ✗ Safety summary line. |
tools.maxCalls |
number |
Total tool calls stayed at or below the cap. |
order |
string[] |
The listed tool names appear as a subsequence of the actual calls (gaps allowed). |
artifacts[].path |
string |
Sandbox-relative path; globs supported. |
artifacts[].exists |
boolean (default true) |
The file is (or is not) there after the run. |
artifacts[].matches |
regex | Multiline regex matches the file contents. |
artifacts[].notMatches |
regex | ...and this one does not. |
artifacts[].equalsFile |
path | Byte-for-byte equal to a golden file, after trailing-whitespace normalisation. |
artifacts[].minBytes |
number |
Guards against a technically-present, actually-empty file. |
output.schema |
path to JSON Schema | The final message parses as JSON (a fenced json code block is tolerated) and validates. |
output.contains |
string[] |
Case-insensitive substrings, all present. |
output.notContains |
string[] |
...none present. |
output.matches |
regex | Regex over the final message. |
output.rubric |
natural language | Judged by an LLM. Opt-in (--judge + ANTHROPIC_API_KEY); skipped as a warning otherwise, never a silent pass or a hard fail. |
output.evaluator |
evaluator name | Your own JS function decides. See custom evaluators. |
budget.tokens |
number |
Total tokens stayed under. Missing usage → skipped with a warning, not a fake pass. |
budget.usd |
number |
Same, in dollars. |
budget.wallMs |
number |
Same, in wall-clock milliseconds. |
exitCode |
number (default 0) |
Strict equality against the agent's exit code. |
A case passes when no assertion failed at severity: "error". Warnings (missing usage, an
unconfigured judge) are visible but never turn a run red on their own.
One skillproof.yaml sits next to each SKILL.md:
version: 1
skill: ./SKILL.md
defaults:
agents: [mock]
fixture: ./fixtures/basic
timeoutMs: 120000
budget: { tokens: 20000, usd: 0.10 }
cases:
- name: activates on a pdf extraction request
prompt: Extract the tables from report.pdf into out/tables.csv
expect:
activation: required
tools:
allowed: [Read, Write, Bash]
required: [Read]
forbidden: [WebFetch]
order: [Read, Write]
artifacts:
- path: out/tables.csv
matches: "^col_a,col_b"
output:
contains: ["tables"]
budget: { tokens: 15000 }
- name: does not activate on unrelated chit-chat
prompt: What is the capital of France?
expect:
activation: forbiddenTwo rules worth internalising:
defaults.expectis shallow-merged per top-level key — a case that setstools:replaces the defaulttools:block wholesale, it does not merge inside it.- Unknown keys are a validation error.
forbiden:fails the run instead of quietly asserting nothing, which is the entire point.
Discovery: any skillproof.yaml, skillproof.yml, or *.skilltest.yaml. Any SKILL.md
with no manifest is reported as skipped, "no tests" — an untested skill is a finding,
not an absence.
name: SkillProof
on: pull_request
permissions:
contents: read
pull-requests: write
jobs:
skills:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: skillproof/skillproof/action@v1
with:
path: ./skills
agents: mock
mode: replayInputs: path, agents, mode, comment, fail-on-regression, baseline-artifact,
github-token. Outputs: passed, failed, total, report-path, badge-path.
The action runs the suite, writes skillproof-run.json, appends the report to the job
summary, and upserts a single PR comment (marker: <!-- skillproof-report -->) so your
PR does not accumulate twelve bot comments.
<!-- skillproof-report -->
### SkillProof — 18/20 passed
```text
SkillProof: 18/20 passed
✓ Activation precision: 96%
✓ Artifact contract: passed
✗ Safety: attempted forbidden shell command
⚠ Token cost increased 42% vs baseline
```
| Skill | Case | Agent | Result | Tokens |
| --- | --- | --- | --- | --- |
| pdf-extract | activates on a pdf extraction request | mock | pass | 1200 |
| pdf-extract | blocks forbidden shell command | mock | fail | 900 |
<details>
<summary>2 failing assertions</summary>
**pdf-extract › blocks forbidden shell command** (`mock`)
- ✗ `tools.forbidden` — expected no call to `Bash`, got Bash("rm -rf /")
- expected `no call to Bash`, actual `Bash("rm -rf /")`
</details>
_20 cases in 4.1s · mode: replay · agents: mock · branch: fix-pdf-tables · skillproof v0.1.0_npx skillproof badge --from skillproof-run.json --out badge.jsonbadge.json is a shields.io endpoint payload
(green at 100%, yellow from 80%, red below). Commit it, or publish it from the action, then:
npx skillproof matrix ./skills --agents claude-code,codex,gemini,opencode| Skill | claude-code | codex | gemini | opencode |
| --- | --- | --- | --- | --- |
| pdf-extract | ✅ | ✅ | ⚠️ | ✅ |
| sql-migration | ✅ | ✅ | ✅ | ❌ |
| k8s-manifest | ✅ | ⚠️ | – | – |
✅ all cases pass · ⚠️ partial · ❌ none pass · – not runPaste it straight into your skill's README. It is the answer to "does this work on my agent?", and it is generated rather than claimed. See docs/adapters.md for exactly what each adapter can and cannot observe — the honest version, including the ones with no reliable token accounting.
Real agent runs cost money, need credentials, and are not reproducible. SkillProof records them once and replays them forever.
skillproof test ./skills --agents claude-code --mode record # spend money once
git add .skillproof/cassettes
skillproof test ./skills --agents claude-code --mode replay # free, offline, deterministic| Mode | Behaviour |
|---|---|
auto (default) |
Replay on a cassette hit; run live and record on a miss. |
record |
Always run live, always overwrite the cassette. |
replay |
Must hit. A miss is an error, never a silent live call. Use this in CI. |
live |
Never read or write cassettes. |
A cassette stores the whole trace and the artifacts the agent wrote; on replay the
artifacts are restored into the sandbox before assertions run, so artifact expectations are
exercised for real. The cassette key is a hash of the agent id, the skill's content hash,
the prompt, the fixture hash and the adapter version — edit the skill and the cassette
stops matching, which is exactly what you want. Default directory:
.skillproof/cassettes.
The mock adapter needs no cassette at all: it is scripted, in-process, offline, and
deterministic (fixed wallMs, tool-call timestamps at index * 10). Every example in this
repo runs green with --agents mock --mode live and no network.
When contains and a regex are not enough, write a function:
// evaluators/csv.mjs
export default [
{
name: "valid-csv",
async evaluate(ctx) {
const csv = ctx.artifacts.get("out/tables.csv") ?? "";
const widths = new Set(csv.trim().split("\n").map((l) => l.split(",").length));
return {
id: "custom.valid-csv",
kind: "custom",
ok: widths.size === 1,
severity: "error",
message:
widths.size === 1
? "every CSV row has the same column count"
: `ragged CSV: row widths ${[...widths].join(", ")}`,
};
},
},
];evaluators: [./evaluators/csv.mjs]
cases:
- name: emits a well-formed csv
prompt: Extract the tables from report.pdf into out/tables.csv
expect:
output: { evaluator: valid-csv }The evaluator receives the resolved case, the full agent trace, the sandbox path, the
artifact map and the loaded evaluator registry, and returns one AssertionResult or an
array of them. Modules are plain ESM, resolved relative to the manifest.
skillproof test [paths...] # default path: "."
--agents <a,b> default: every available adapter, falling back to mock
--mode <auto|record|replay|live> default auto
--cassettes <dir> default .skillproof/cassettes
--reporter <tty|json|markdown|junit> repeatable, default tty
--out <file> write the last non-tty reporter's output here
--baseline <file> a previous run.json -> enables the regression section
--filter <substr>
--concurrency <n> default 4
--bail
--judge enable the LLM rubric judge (needs ANTHROPIC_API_KEY)
--keep-sandbox
--quiet / --verbose
exit 1 on any failure, 2 on usage error
skillproof init [skillDir] # scaffold skillproof.yaml + fixtures/basic + mock.agent.json
skillproof list [paths...] # table of skills, #cases, allowed-tools?, license?
skillproof matrix [paths...] # skill x agent pass matrix -> markdown
skillproof badge --from run.json [--out badge.json]
skillproof compare --base a.json --head b.json [--markdown]
skillproof doctor # which agent CLIs are installed
Every run happens in a disposable sandbox directory: the fixture is copied in, the skill is
installed under the agent's own convention (.claude/skills/, .codex/skills/, ...), the
agent is pointed at the sandbox as its cwd, and the whole thing is deleted afterwards
unless you pass --keep-sandbox.
- Not a linter. It does not have opinions about your markdown, your heading levels, or your frontmatter style. It runs the skill and checks what happened. (If it never activates, that is a behavioural finding, not a style one.)
- Not a security scanner.
tools.forbiddenasserts that your test prompt did not provoke a forbidden tool call. It is a regression guard, not a proof of safety, not a sandbox escape audit, and not a substitute for actually restricting what your agent can do. A green run means "this prompt behaved"; it does not mean "this skill is safe". - Not a package manager. It does not install, publish, resolve, version, or distribute
skills. Bring your own directory of
SKILL.mdfiles. - Not an eval harness for models. The unit under test is the skill. Swapping the model is a variable you control, not the thing being scored.
- docs/manifest-reference.md — every manifest key, type, default and semantics.
- docs/adapters.md — per-agent capabilities, honestly, and how to write a new adapter.
- CONTRIBUTING.md — repo layout, tests, and the rule that
SPEC.mdis the contract. - examples/skills/ — ten worked skills, each with a positive activation case, a negative activation case and a safety case.
MIT licensed.