About klbench
What runs well on Tanzu Platform?
klbench is an open leaderboard of LLMs running on the Tanzu GenAI tile, scored on what coding agents actually do: tool calls, structured output, debugging, multi-turn context, platform operations, and end-to-end agentic tasks run under two agent harnesses (opencode and a custom harness) so the spread between them is itself a measurement.
Unlike cloud-only benchmarks, the same model behaves very differently depending on the hardware and engine underneath it. This site surfaces those differences as first-class filters — and correct-but-too-slow answers score zero, which keeps CPU-vs-GPU comparisons honest.
This site is hosted on the NDC foundation but displays results from every foundation that contributes data (CDC, DEV210, TDC, and others). Hosting location is unrelated to data provenance.
Test suite
148 YAML-defined tests across 18 categories: basic, tool_use, structured_output, coding, debugging, long_context, instruction, file_ops, multi_turn, reasoning, writing, research, monitoring, iac, ci_repair, repo_patch, sysadmin, and agentic — 46 of them tier:hard. Grading is deterministic wherever objectively checkable (execution checks, schema validation, graded partial credit); reasoning, writing, and research use an LLM judge from a different model family with per-criterion rubrics, K-vote shuffling, and a recorded judge fingerprint. Endpoint-served models use opencode plus a custom loop; subscription targets use their matching Claude Code or Codex harness. Agentic rows are scored per task rather than per framework so the fan-out cannot outweigh the rest of the suite.
Items are held to a published design contract, enforced in CI: the answer may not appear in the prompt, a diagnosis may not be graded by keyword presence, every rubric criterion must be failable, an item needs at least three independently-failing assertions, and a check must fail against an untouched fixture. On the current panel 99% of scored rows discriminate between models and 80% sit in the 30–70% difficulty band.
How scoring works
- The headline follows the hardware class — GPU and cloud boards use the category-balanced capability composite, while CPU boards use usability: the equally weighted per-test mean with every timed-out task counted as zero. Both are shown because they answer different questions and can differ even without timeouts. Each test runs 3× by default; the recorded score is the median, with a 95% category-clustered bootstrap CI.
- Completion rate — the share of tests that returned within the latency budget. Correct-but-too-slow answers score zero, so a model that only finishes a fraction of the suite is scored on the whole suite, not the fraction it managed. GPU and CPU models are ranked in separate groups because they run different subsets and have different latency realities.
- Statistical tiers (T1, T2…) — models are compared by paired per-test deltas, since every model answers the same tests. When the paired 95% CI crosses zero the suite cannot rank them: they share a tier. Rank order inside a tier is display order, not a measured difference.
- Minimum detectable gap — each result states the smallest composite difference the suite can resolve at its run count. Gaps below it are reported as ties, never as wins.
- Judge debiasing — where a human-labeled calibration exists for the judge, judged category scores get the Rogan-Gladen correction (raw scores are kept alongside).
- Difficulty calibration — test difficulty is measured against the live model panel, not asserted; saturated tests are hardened or retired, and a contamination canary is embedded in every test file.
How the score is weighted
- Composite — the equal mean of 18 category means. Every category is worth 1/18 regardless of how many items it holds, so one item is worth 0.37% of the composite in the 15-item coding category and 2.78% in the 2-item research category.
- Agentic and repo_patch fan out to 2 harness rows per task; the rows are averaged per task before the category mean, so the fan-out does not add weight.
- Timeouts score 0 everywhere.
- Refusals are excluded from the capability composite and count as 0 in usability and in the effective score. The refusal count is shown beside every row.
- CPU and entry-GPU tiers rank by usability (equal per-item mean, timeouts and refusals as 0); mid/high GPU and subscription tiers rank by the composite.
- Judge-scored categories — reasoning, writing and research — are 3 of 18 and are reported uncorrected unless a human-labeled judge calibration exists for the run.
Two boards: Ships and Thinks
The headline is Ships: Ships = mean of per-PRD mean scores over the non-saturated, mechanically graded PRDs, run under the opencode on-prem harness on placement-attested workers. Means, not medians: PRD scores are coarse and bimodal, so a median can hide a 40% failure rate behind a 1.0. Saturated PRDs (every model mean ≥ 0.95) are excluded, not hidden. Provisional until every model × PRD cell has at least two runs in one era. The capability suite is Thinks, the single-shot diagnostic behind it. They are never blended, because on this panel the capability composite correlates negatively (ρ ≈ −0.6, n = 6) with real-work outcome; it is a diagnostic, not a ranking — a model can ace the exam and still not ship. Rows without eligible PRD runs (CPU lanes, new models) show no headline and sort after ranked rows. Ranks on the Ships board share a number when the paired gap over PRDs is below the minimum detectable effect at power 0.80 (≈ 0.16 at seven PRDs).
How judging works
Only the ten writing, research, reasoning and format-conflict items are graded by a language-model judge; every other score is executable or pattern-graded. Judged items are scored by a two-vendor panel (OpenAI Codex and Anthropic Claude CLIs), three votes per item with the criteria shuffled per vote, and the per-criterion median is taken. Each result records which member cast each vote and whether any vote fell back to the other vendor. Disclosed overlap: one panel member (gpt-oss-120b) shares a vendor with one judge — the judged share of its composite is small, and a cross-judge bias study is published alongside the boards. Thinking parity: from the v5 era the harness requests reasoning_effort medium on every request; earlier eras used each engine's default, which is disclosed per row (a "no-think" chip marks models that only think when asked).
| category | items | per-item weight |
|---|---|---|
| coding | 15 | 0.37% |
| instruction | 13 | 0.43% |
| iac | 12 | 0.46% |
| agentic | 12 | 0.46% |
| monitoring | 12 | 0.46% |
| structured_output | 11 | 0.51% |
| tool_use | 10 | 0.56% |
| reasoning | 8 | 0.69% |
| debugging | 8 | 0.69% |
| multi_turn | 8 | 0.69% |
| repo_patch | 7 | 0.79% |
| ci_repair | 7 | 0.79% |
| sysadmin | 6 | 0.93% |
| long_context | 6 | 0.93% |
| file_ops | 3 | 1.85% |
| writing | 3 | 1.85% |
| basic | 3 | 1.85% |
| research | 2 | 2.78% |
Data pipeline
Every result is generated by the klbench runner (klbench run) and lands as a JSON file in results/<foundation>/<hardware>/ that must validate against schema/result-v3.schema.json. CI runs tools/validate.py on every pull request — invalid data cannot land — and the leaderboard rebuilds on merge.
How to contribute
- Clone the repo:
git clone https://github.com/nkuhn-vmw/klbench - Run
klbench run --url <endpoint> --model <name> --foundation <name> --hardware cpu|gpu - Validate:
python3 tools/validate.py results/ - Commit the generated JSON under
results/<foundation>/<hardware>/ - Open a pull request — CI validates the schema, the leaderboard rebuilds on merge