Agent-ready playbooks for LLM serving benchmarks, capacity planning, torch-profiler triage, pipeline analysis, compute simulation, SGLang/vLLM optimization, human code review, production incidents, and model PR intelligence.
This repository is built for AI infrastructure engineers who want agents to do real work, not recite generic prompts.
It gives an agent the operational memory needed to benchmark SGLang, vLLM, TensorRT-LLM, and TokenSpeed fairly; explain serving capacity from startup logs; split prefill and decode profiler evidence; inspect traces at layer and kernel level; estimate operator FLOPs and MFU; review SGLang patches against real maintainer discussion patterns; run Humanize-governed SGLang and vLLM SOTA loops; triage SGLang production incidents from a replay; and keep model-family optimization history close to the code that actually changed.
For standalone kernel campaigns and kernel evidence tools, see the sibling project KDA-Pilot.
If this saves you one stale model-support assumption, one misleading profiler trace, or one late-night benchmark loop, a star helps more AI-infra engineers find it.
| Skill | Use it when |
|---|---|
llm-serving-auto-benchmark |
You need a fair, bounded serving benchmark search for SGLang, vLLM, TensorRT-LLM, TokenSpeed, or another OpenAI-compatible stack. |
llm-serving-capacity-planner |
You need to explain SGLang or vLLM startup memory, KV cache budget, request capacity, or OOM pressure from logs. |
llm-torch-profiler-analysis |
You need a three-table profiler report that keeps extend/prefill and decode evidence separate. |
llm-pipeline-analysis |
You need forward-pass, layer, and kernel-level timing from a torch profiler trace, including anchor boundaries and Perfetto ranges. |
model-compute-simulation |
You need operator shapes, FLOPs, MFU estimates, kernel-to-op mapping, or parallelism what-if analysis for an LLM serving shape. |
model-pr-diff-dossier |
You need to create or revise model PR history docs with manual diff-reviewed cards instead of shallow PR-title summaries. |
sglang-humanize-review |
You need SGLang code-review findings grounded in full human PR review episodes from project start through the latest refresh (June 2026), including inline code context, top-level discussion, review summaries, and multi-round replies. Every review opens with a PR comprehension pass — a change summary plus a Mermaid execution flowchart with the diff's modified steps marked — so the reviewer sees how the PR runs before the findings. |
sglang-sota-humanize-loop |
You want one model-level Humanize RLCR loop that owns SGLang gap decisions against a selected comparison framework set, profiler triage, required layer-pipeline deep dives, SGLang patches, optional ncu-report-skill evidence, and real-model revalidation after the fixed fair benchmark. |
vllm-sota-humanize-loop |
You want one model-level Humanize RLCR loop that owns gap decisions, profiler triage, required layer-pipeline deep dives, vLLM patches, optional ncu-report-skill evidence, and real-model revalidation after the fixed fair benchmark. |
sglang-prod-incident-triage |
You need to turn queue growth, timeouts, wrong outputs, crashes, or distributed stalls into a replay and next debug step. |
model-architecture-diagram |
You need original public architecture diagrams for popular LLM, VLM, MoE, OCR, and diffusion model families. |
sglang-sota-humanize-loop always patches SGLang, while the competitor set is
caller-controlled. By default the comparison framework set can include vLLM,
TensorRT-LLM, and TokenSpeed; a prompt can also narrow it, for example:
Use sglang-sota-humanize-loop for <model>.
comparison_frameworks: [vllm]
Do not consider TensorRT-LLM or TokenSpeed; record them as user-excluded.
The model optimization layer is now one knowledge base:
model-pr-optimization-history. It contains
66 PR-driven history dossiers and a small query helper. These are not
per-model runbook skills; they preserve diff-backed model evolution records for
SGLang, vLLM, TensorRT-LLM, and TokenSpeed so SOTA loops can read prior source
and PR evidence before patching.
| Framework | PR histories |
|---|---|
| SGLang | 31 |
| vLLM | 31 |
| TensorRT-LLM | 2 |
| TokenSpeed | 2 |
Covered families include:
DeepSeek V3/R1/V3.1/V3.2/V4, Qwen3, Qwen3-Coder, Qwen3-Next,
Qwen3.5/Qwen3.6, Qwen VLM/Omni/ASR, GLM 4.5/4.6/4.7/5,
Kimi, MiniMax, Llama 4, Mistral Small 4, Mixtral, Nemotron,
Gemma, Ernie 4.5, Intern-S1, InternVL, Hunyuan, MOSS-VL,
GPT-OSS, Step 3.5, Mimo, and model-specific MoE/quantization paths.
Each model-family history is designed to answer practical questions:
- Which PRs changed this model path?
- Was the PR merged, closed, or still open?
- Which files and symbols moved?
- What optimization or correctness risk should be checked before touching it?
- Which upstream idea should be compared before writing a new kernel or fusion?
Query examples:
cd model-pr-optimization-history
python3 scripts/query.py --list
python3 scripts/query.py --framework sglang --model qwen3-core --paths-only
python3 scripts/query.py --framework vllm "qwen3 fused qk norm"
python3 scripts/query.py --framework tokenspeed --model qwen35 qk rmsnormOpen PR freshness is tracked separately from merged history cards:
python3 tools/check_open_pr_watch.py --format markdown \
--output model-pr-optimization-history/open-pr-watch.mdThis report uses the GitHub pulls API with an anonymous REST fallback when
gh api is rate-limited. If every repo fetch fails, the tool exits non-zero
instead of writing a misleading empty report.
The repo is opinionated about evidence because performance work gets noisy fast.
- Benchmark rows should include model, framework, GPU count, workload, request rate or concurrency, SLA status, launch command, benchmark command, and raw artifacts.
- Profiler reports should keep prefill and decode separate, then emit the same three tables: kernel table, overlap-opportunity table, and fuse-opportunity table.
- SOTA claims should be scoped to the exact model, hardware, framework commits, precision, workload, and SLA used in the run.
- SGLang human review should use the full PR episode corpus: inline review threads for line-local findings, PR conversations for design/test/repro negotiation, and review submissions for blocking maintainer summaries.
- Humanize SOTA loops should keep only the fixed fair benchmark outside the patch loop; gap decisions, profiler triage, required layer-pipeline deep dives, kernel evidence, target-framework code changes, and revalidation all stay inside one model-level RLCR loop.
- Kernel-local fixes inside that loop should use
ncu-report-skillwhen Nsight Compute counter evidence is needed, store NCU digests, and still pass the same real-model benchmark/profile gate. - Incident triage should start from replayable evidence instead of changing code from symptoms alone.
- Model optimization histories should point back to PRs, files, diffs, and risk surfaces rather than vague summary text; they live as one PR-driven knowledge base, not per-model skills.
- Root-level
update_prompt.mdcaptures the full refresh and validation workflow for updating this repo again without relying on memory from a previous run.
This repository is not Codex-only. The skills are plain SKILL.md directories
and can be installed into Claude Code, Codex, Kimi, or another compatible agent
runtime.
The repository ships a .claude-plugin/ manifest so the whole skill set can be
installed as a single Claude Code plugin via the built-in marketplace flow:
/plugin marketplace add BBuf/AI-Infra-Auto-Driven-SKILLS
/plugin install ai-infra-auto-driven-skills@ai-infra-auto-driven-skills
/reload-plugins
After reload, the 12 skills appear namespaced as
ai-infra-auto-driven-skills:<skill-name> (for example
ai-infra-auto-driven-skills:sglang-sota-humanize-loop). Update later with
/plugin marketplace update ai-infra-auto-driven-skills.
Prefer this when you only want a subset of the skills, or when developing against a local checkout. Symlink is recommended for local development because updates to this checkout are picked up immediately:
git clone https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS.git
cd AI-Infra-Auto-Driven-SKILLS
mkdir -p ~/.claude/skills
ln -s "$PWD/skills/llm-serving-auto-benchmark" ~/.claude/skills/llm-serving-auto-benchmark
ln -s "$PWD/skills/llm-serving-capacity-planner" ~/.claude/skills/llm-serving-capacity-planner
ln -s "$PWD/skills/llm-torch-profiler-analysis" ~/.claude/skills/llm-torch-profiler-analysis
ln -s "$PWD/skills/llm-pipeline-analysis" ~/.claude/skills/llm-pipeline-analysis
ln -s "$PWD/skills/model-compute-simulation" ~/.claude/skills/model-compute-simulation
ln -s "$PWD/skills/model-optimization/model-pr-diff-dossier" ~/.claude/skills/model-pr-diff-dossier
ln -s "$PWD/skills/sglang-humanize-review" ~/.claude/skills/sglang-humanize-review
ln -s "$PWD/skills/sglang-sota-humanize-loop" ~/.claude/skills/sglang-sota-humanize-loop
ln -s "$PWD/skills/vllm-sota-humanize-loop" ~/.claude/skills/vllm-sota-humanize-loop
ln -s "$PWD/skills/sglang-prod-incident-triage" ~/.claude/skills/sglang-prod-incident-triage
ln -s "$PWD/skills/model-architecture-diagram" ~/.claude/skills/model-architecture-diagram
ln -s "$PWD/model-pr-optimization-history" ~/.claude/skills/model-pr-history-knowledgeRestart Claude Code after installing. The skills can then be invoked by name,
for example [$llm-serving-auto-benchmark],
[$llm-serving-capacity-planner], [$llm-torch-profiler-analysis],
[$llm-pipeline-analysis], [$model-compute-simulation],
[$model-pr-diff-dossier], [$sglang-humanize-review],
[$sglang-sota-humanize-loop], or [$vllm-sota-humanize-loop].
If you prefer copies instead of symlinks, replace ln -s with cp -R. Copy
model-pr-optimization-history only when you want the agent to query the
PR-driven model knowledge base locally. It replaces the old per-model runbook
skill layout with one shared knowledge root.
For Codex, Kimi, or another compatible runtime, copy or symlink the same directories into that runtime's skill directory:
cp -R skills/llm-serving-auto-benchmark <agent-skill-dir>/llm-serving-auto-benchmark
cp -R skills/llm-serving-capacity-planner <agent-skill-dir>/llm-serving-capacity-planner
cp -R skills/llm-torch-profiler-analysis <agent-skill-dir>/llm-torch-profiler-analysis
cp -R skills/llm-pipeline-analysis <agent-skill-dir>/llm-pipeline-analysis
cp -R skills/model-compute-simulation <agent-skill-dir>/model-compute-simulation
cp -R skills/model-optimization/model-pr-diff-dossier <agent-skill-dir>/model-pr-diff-dossier
cp -R skills/sglang-humanize-review <agent-skill-dir>/sglang-humanize-review
cp -R skills/sglang-sota-humanize-loop <agent-skill-dir>/sglang-sota-humanize-loop
cp -R skills/vllm-sota-humanize-loop <agent-skill-dir>/vllm-sota-humanize-loop
cp -R skills/sglang-prod-incident-triage <agent-skill-dir>/sglang-prod-incident-triage
cp -R skills/model-architecture-diagram <agent-skill-dir>/model-architecture-diagram
cp -R model-pr-optimization-history <agent-skill-dir>/model-pr-history-knowledgeThese skills are exercised with coding agents in full-autonomy mode. For reproducibility, here is exactly how I launch them.
Claude Code — Opus 4.8 at max effort with Auto mode (the "Effort (Max)" + "Auto mode" toggles), i.e. auto/bypass-permission so the agent runs unattended:
claude --permission-mode bypassPermissions --model opus --effort maxUltracode mode — the maximum-thoroughness setting (the "Effort (Ultracode –
xhigh + workflows)" entry in the effort menu, paired with Auto mode). Ultracode
is not a launch-flag effort value: claude --effort ultracode warns (Unknown --effort value 'ultracode') and falls back to the default — the valid --effort
flag levels are low, medium, high, xhigh, max. It is a composite of xhigh
effort + dynamic workflows enabled, so the agent reasons at xhigh and authors
multi-agent workflows on substantive tasks. Enable dynamic workflows once — the
Dynamic workflows toggle in /config (settings key enableWorkflows) — then
launch at xhigh:
claude --permission-mode bypassPermissions --model opus --effort xhighOr in one self-contained command:
claude --permission-mode bypassPermissions --model opus --effort xhigh --settings '{"enableWorkflows": true}'With workflows enabled, the in-session /effort menu shows "Ultracode"; to opt a
single prompt in instead, include the keyword ultracode in that message.
Codex — full-access, no approval prompts:
codex --yolo --sandbox danger-full-access --ask-for-approval neverBoth run unsandboxed / auto-approve because the work happens against isolated checkouts with their own benchmark + correctness gates.
skills/
├── llm-serving-auto-benchmark/ # serving benchmark search and comparison
├── llm-serving-capacity-planner/ # startup memory and request capacity analysis
├── llm-torch-profiler-analysis/ # profiler capture and trace triage
├── llm-pipeline-analysis/ # forward/layer/kernel trace analysis
├── model-compute-simulation/ # operator FLOPs, tensor shapes, and MFU
├── sglang-humanize-review/ # human SGLang PR review corpus and workflow
├── sglang-sota-humanize-loop/ # Humanize-governed SGLang SOTA loop
├── vllm-sota-humanize-loop/ # Humanize-governed vLLM SOTA loop
├── sglang-prod-incident-triage/ # replay-first serving incident workflow
├── model-architecture-diagram/ # public architecture diagram resolver
└── model-optimization/
└── model-pr-diff-dossier/ # shared PR history quality standard
model-pr-optimization-history/
├── SKILL.md # knowledge-base usage instructions
├── scripts/query.py # local model/keyword query helper
├── sglang/ # 31 PR-driven SGLang model histories
├── vllm/ # 31 PR-driven vLLM model histories
├── tensorrt_llm/ # TensorRT-LLM competitor histories
└── tokenspeed/ # TokenSpeed competitor histories
prompts/
├── sglang-sota-b200-prompts.md # B200 SGLang SOTA task prompts
├── sglang-sota-b200-codex-goal-prompts.md
├── sglang-sota-h200-prompts.md # H200 SGLang SOTA task prompts
└── sglang-sota-h200-codex-goal-prompts.md
- Humanize provides the RLCR workflow that powers the Humanize-governed SGLang and vLLM SOTA loops.
- KDA-Pilot is the sibling home for standalone kernel loops, kernel knowledge, and NCU report workflows.