openbench-runner

A small, configurable wrapper around OpenBench for running:

• GPQA Diamond against an OpenRouter, vLLM, or other OpenAI-compatible target
• TAU-bench Airline against the verified task data, with a separately configurable user simulator
• JSONSchemaBench for structured-output (response_format: json_schema) reliability
• An open-loop uptime load generator (uptime.py) that measures uptime under Poisson traffic — separate from OpenBench (see Uptime load generator)

The goal is to make serving-stack comparisons repeatable using the benchmarks described in OpenRouter's Auto Exacto announcement. It is not a claim that a local run is bit-for-bit identical to OpenRouter's published evaluation.

After each run, run.py automatically post-processes the .eval log with analyze.py into the per-provider reliability metrics OpenRouter publishes but OpenBench does not report: tool call error rate (from a tau run), structured output error rate (from a jsonschema run), and cache hit rate. See Derived metrics.

Setup

Requirements: Git, uv, and Python 3.12.

git clone https://github.com/moreh-dev/openbench-runner.git
cd openbench-runner
uv sync --frozen
cp .env.example .env
cp config.example.yaml config.yaml

For TAU-bench, install the pinned verified data snapshot:

./scripts/setup_tau_verified.sh

Edit .env for API keys and config.yaml for the target model and evaluation settings. Neither file is committed.

The lock file fixes OpenBench 0.5.3, Inspect AI 0.3.125, and TAU2 commit 558e6cd066d7bf05db587fa2dc1509765c7d03bc. The setup script fixes the verified dataset at commit 9d0d5ae0a5a80007ec15083448350bb9f8f1cc00 and checks the Airline task file's checksum.

Configure a target

target.provider selects the OpenBench model adapter. It is separate from an OpenRouter inference provider such as Alibaba or Together.

Direct vLLM:

target:
  provider: vllm
  model: vendor/model-name
  base_url: http://localhost:8000/v1

OpenRouter:

target:
  provider: openrouter
  model: vendor/model-name
  base_url: https://openrouter.ai/api/v1

Set OPENROUTER_API_KEY in .env. To pin an OpenRouter inference provider and disable fallback, add request arguments supported by OpenRouter:

target:
  provider: openrouter
  model: vendor/model-name
  base_url: https://openrouter.ai/api/v1
  model_args:
    only: [Alibaba]
    allow_fallbacks: false

Any OpenAI-compatible gateway:

target:
  provider: openai-api
  service: onprem
  model: gateway-model-name
  base_url: https://llm-gateway.example.com/v1

For openai-api, service is a namespace used by Inspect AI to find credentials and the optional base URL. service: onprem reads ONPREM_API_KEY (and, when not supplied in YAML, ONPREM_BASE_URL); hyphens become underscores in environment-variable names. The service segment is removed from the model value sent to the API. Use a dummy key if the server does not authenticate. service is not needed for vllm or openrouter.

Moreh TaaS (a LiteLLM gateway) is an openai-api target. Set TAAS_API_KEY in .env:

target:
  provider: openai-api
  service: taas
  model: gemma-4-31b-it-anthropic
  base_url: https://taas.staging.moreh.dev/v1

All choices call /v1/chat/completions. Prefer vllm for a direct vLLM server because OpenBench adds vLLM-specific message handling; use openai-api for a generic gateway implementing the OpenAI-compatible chat API.

See config.example.yaml for all available settings.

Run

Preview the generated OpenBench command without making requests:

uv run python run.py gpqa --dry-run
uv run python run.py tau --dry-run
uv run python run.py jsonschema --dry-run

Run an evaluation:

uv run python run.py gpqa
uv run python run.py tau
uv run python run.py jsonschema

The most common settings can also be changed without editing YAML:

uv run python run.py gpqa \
  --provider vllm \
  --model vendor/model-name \
  --base-url http://localhost:8000/v1

Available overrides are shown by uv run python run.py --help. In particular, --limit 1 --epochs 1 is useful for a smoke test. The YAML file remains the recommended record for a reportable run.

Open the result viewer:

uv run bench view

OpenBench writes detailed sample logs under logs/. Each run gets its own subdirectory (logs/<benchmark>-<timestamp>-<pid>/) so that benchmarks started concurrently keep their logs — and their derived-metric analysis — separate. Keep those logs together with the exact config, model revision, serving arguments, and hardware used for a comparison.

What is evaluated

Preset	OpenBench task	Samples	Important defaults
gpqa	gpqa_diamond	198	Temperature 0.5; 10 epochs and mean reduction in the example config
tau	tau_bench_airline	50 verified tasks	Temperature 0; max_steps: 200; user simulator configurable independently from the target
jsonschema	jsonschemabench	~10K schemas (use subset/limit)	Temperature 0; parity defaults use_response_schema: true, strip_markdown: false, strict_mode: false

The jsonschema preset defaults prioritize parity with OpenRouter's per-provider structured-output error rate:

• use_response_schema: true sends response_format: {type: json_schema} (the path OpenRouter measures) instead of only prompting for JSON.
• strip_markdown: false counts a ```json fenced reply as invalid JSON, matching a raw JSON.parse.
• strict_mode: false (the default) sends the dataset schemas exactly as authored (partial-required, unmodified). Set it to true to apply the stricter OpenAI structured-output contract — which sets both the response_format strict flag and the schema adaptation (all properties required + additionalProperties: false); the two always move together.

The full JSONSchemaBench mix is large; set jsonschema.subset (e.g. Github_easy, Kubernetes) and/or jsonschema.limit for a cheaper run, and keep common.max_tokens generous so long instances are not truncated into invalid JSON. The dataset is fetched from Hugging Face on first use, so run it once on a host with huggingface.co access (or pre-populate the HF cache).

TAU's max_steps counts user-model, target-model, and environment/tool steps. It is not the number of dialogue turns. The runner adds OpenBench's required --alpha flag and sets TAU2_DATA_DIR from tau.data_dir.

For the closest comparison with OpenRouter's current setup, configure TAU's user simulator as openrouter/openai/gpt-5.1 and set OPENROUTER_API_KEY. This OpenRouter model is used only for the simulated user; the target can remain local.

reasoning_effort: null deliberately omits that request option because OpenRouter has not published the value used for its provider scores. If you set an effort level, use the same explicit level for every target and report it with the result.

Use limit: 1 only as a connectivity smoke test. Reportable scores should use the full dataset and the same epoch count and generation settings on every serving stack.

Derived metrics

OpenBench reports task accuracy. analyze.py reads the .eval logs a run produces and derives the per-provider reliability metrics OpenRouter aggregates but OpenBench does not report. run.py runs it automatically after a successful evaluation, analyzing only that run's own log (from its per-run subdirectory), so it is safe to run several benchmarks at once; pass --no-analyze to skip it. You can also run it directly over any log(s) later (the directory form recurses into the per-run subdirectories):

uv run python analyze.py logs/                 # every .eval log in a directory
uv run python analyze.py logs/<run>.eval       # one log
uv run python analyze.py logs/ --json out.json # also write full results as JSON

Metric	From	Definition used
tool call error rate	a tau run	Request-level errored / requests_with_tool_calls; a request is errored if any returned tool call is InvalidJson (unparsable arguments), UnknownName (name not in the request's tools), or SchemaMismatch (arguments fail the tool's parameter schema). Matches OpenRouter's definition.
structured output error rate	a jsonschema run	errored / all json_schema requests, where errored = API rejection or invalid JSON or schema violation. Re-validated with a Draft 7 validator (plus a syntax-only and a schema-only breakdown) for parity with OpenRouter's @cfworker checker, rather than OpenBench's built-in Draft 2020-12 scorer.
cache hit rate	any run	Token-weighted sum(cache_read_tokens) / sum(input_tokens) (usage.prompt_tokens_details.cached_tokens); reports n/a when the endpoint does not return cached-token counts.

Only the target model's calls are counted; a run's other models (for example TAU's user simulator) are excluded. These numbers approximate OpenRouter's published values but are not identical: OpenRouter measures real edge traffic, and the validator draft, markdown handling, schema adaptation, and max_tokens all shift the result. Report them for relative serving-stack comparison, not as absolute equality with a public score.

Uptime load generator

uptime.py is a separate tool from the OpenBench presets. OpenBench (via Inspect AI) is closed-loop — a fixed number of samples bounded by a concurrency cap — so it cannot create the queueing/overload that uptime failures come from. uptime.py is open-loop: it fires requests on an independent arrival process for a fixed duration, so the endpoint can actually saturate. It measures OpenRouter's definition:

uptime = successful / (successful + derankable-error)     # 429 excluded

where a request is a derankable error on 401/402/404, any 5xx, a connect failure, a stall (no streamed bytes for stall_timeout), or an HTTP 200 whose finish_reason is error (or that carries a mid-stream error); 429 is counted separately (rate_limited); 400/403/413/422 are excluded as user errors. A slow-but-progressing response is a success — the stall timeout is a per-read gap, not a total wall-clock cap, so a long generation that keeps streaming is not penalized (latency is reported separately). This mirrors OpenRouter, where slow-but-successful counts as success and only a fetch-timeout-and-fallback is a failure. The error breakdown labels timeouts distinctly (e.g. stall_timeout, connect_timeout) rather than lumping them under the 200 a stalled stream already returned; the 429 rate and TTFT/E2E latency under load are also reported.

One-time data setup (writes into data/, gitignored):

uv run python scripts/setup_uptime_data.py

This fetches a Gutenberg corpus (prompt bodies) and the Alibaba Qwen-Bailian traceA/coder/thinking production traces (Apache-2.0), which supply realistic (input_length, output_length) pairs reflecting current chat/coding/reasoning traffic.

Run it against the same target: block the presets use:

uv run python uptime.py --dry-run                        # plan + sample lengths, no requests
uv run python uptime.py --rate 5 --duration 300          # 5 req/s Poisson for 5 min
uv run python uptime.py --config configs/taas.yaml --burstiness 0.5   # burstier arrivals

Key uptime: config knobs (see config.example.yaml): rate (requests/sec — the saturation knob; raise it until 429/5xx appear to find the ceiling), duration, burstiness (1.0 = Poisson; <1 burstier), stall_timeout (max seconds between streamed bytes before a request is failed as a stall), max_concurrency (client-side safety cap — a warning prints if it, rather than the server, limits offered load), and workload (source: bailian samples the real traces with a blend of traceA/coder/thinking; source: bimodal is a synthetic fallback needing no download). Each request's length pair is sampled from the trace blend, then the prompt body is filled from the corpus to that input length (the true ISL is recorded from usage.prompt_tokens); arrivals stay Poisson regardless.

Results are written to logs/uptime/uptime_<ts>.jsonl (one row per request) plus a printed summary. Because arrivals are a modeling choice (OpenRouter publishes no sub-daily arrival data) and traffic is generated from our network rather than OpenRouter's edge, treat the number as a relative reliability probe, not an absolute match to a public uptime figure.

Reproduction limits

This repository pins the harness dependencies and the verified TAU data snapshot, and exposes the settings that materially affect requests. Exact equality with a score shown on OpenRouter is still not guaranteed because the following may differ or change:

• model weights, quantization, chat template, tokenizer, and serving-engine revision
• reasoning settings, sampling behavior, maximum output length, and tool-call formatting
• evaluator and user-simulator model revisions
• transient request failures and the samples that were retried
• OpenRouter's evaluation date, repetitions, and aggregation of provider measurements

Treat a difference from the public score as evidence to investigate, not automatically as a serving-performance gap. First verify the model revision and request settings, then repeat the complete evaluation enough times to estimate run-to-run variation.

Troubleshooting

• tau_bench_airline is experimental/alpha: use this runner; it adds --alpha automatically.
• No module named 'tau2': run uv sync --frozen, then invoke the runner through uv run. The dependency lock pins the TAU2 code; TAU2_DATA_DIR only selects its data.
• Wrong or unreachable endpoint: set the base URL to the API root ending in /v1, not /v1/chat/completions. The adapter appends /chat/completions.
• Local vLLM rejects chat requests: this runner uses the chat-completions API. Serve the model with a valid chat template, supplying one explicitly to vLLM when the tokenizer does not include it. The completions and responses endpoints are not selected by this runner.
• JSONDecodeError after some samples: this usually means an HTTP response was truncated or was not JSON. Lower concurrency and keep retry_on_error enabled; inspect the failed sample before comparing scores.
• Timeout behavior: timeout applies to each model API request, not to the complete benchmark run.
• Seats release not implemented for cancellation!!!: this is a known TAU2 Airline environment warning. If the sample completes and is logged, the warning itself does not invalidate the run; keep the same pinned environment version across comparisons.