What runs well on Tanzu Platform?

Test suite

148 YAML-defined tests across 18 categories: basic, tool_use, structured_output, coding, debugging, long_context, instruction, file_ops, multi_turn, reasoning, writing, research, monitoring, iac, ci_repair, repo_patch, sysadmin, and agentic — 46 of them tier:hard. Grading is deterministic wherever objectively checkable (execution checks, schema validation, graded partial credit); reasoning, writing, and research use an LLM judge from a different model family with per-criterion rubrics, K-vote shuffling, and a recorded judge fingerprint. Endpoint-served models use opencode plus a custom loop; subscription targets use their matching Claude Code or Codex harness. Agentic rows are scored per task rather than per framework so the fan-out cannot outweigh the rest of the suite.

Items are held to a published design contract, enforced in CI: the answer may not appear in the prompt, a diagnosis may not be graded by keyword presence, every rubric criterion must be failable, an item needs at least three independently-failing assertions, and a check must fail against an untouched fixture. On the current panel 99% of scored rows discriminate between models and 80% sit in the 30–70% difficulty band.

How scoring works

• The headline follows the hardware class — GPU and cloud boards use the category-balanced capability composite, while CPU boards use usability: the equally weighted per-test mean with every timed-out task counted as zero. Both are shown because they answer different questions and can differ even without timeouts. Each test runs 3× by default; the recorded score is the median, with a 95% category-clustered bootstrap CI.
• Completion rate — the share of tests that returned within the latency budget. Correct-but-too-slow answers score zero, so a model that only finishes a fraction of the suite is scored on the whole suite, not the fraction it managed. GPU and CPU models are ranked in separate groups because they run different subsets and have different latency realities.
• Statistical tiers (T1, T2…) — models are compared by paired per-test deltas, since every model answers the same tests. When the paired 95% CI crosses zero the suite cannot rank them: they share a tier. Rank order inside a tier is display order, not a measured difference.
• Minimum detectable gap — each result states the smallest composite difference the suite can resolve at its run count. Gaps below it are reported as ties, never as wins.
• Judge debiasing — where a human-labeled calibration exists for the judge, judged category scores get the Rogan-Gladen correction (raw scores are kept alongside).
• Difficulty calibration — test difficulty is measured against the live model panel, not asserted; saturated tests are hardened or retired, and a contamination canary is embedded in every test file.

How the score is weighted

• Composite — the equal mean of 18 category means. Every category is worth 1/18 regardless of how many items it holds, so one item is worth 0.37% of the composite in the 15-item coding category and 2.78% in the 2-item research category.
• Agentic and repo_patch fan out to 2 harness rows per task; the rows are averaged per task before the category mean, so the fan-out does not add weight.
• Timeouts score 0 everywhere.
• Refusals are excluded from the capability composite and count as 0 in usability and in the effective score. The refusal count is shown beside every row.
• CPU and entry-GPU tiers rank by usability (equal per-item mean, timeouts and refusals as 0); mid/high GPU and subscription tiers rank by the composite.
• Judge-scored categories — reasoning, writing and research — are 3 of 18 and are reported uncorrected unless a human-labeled judge calibration exists for the run.

Data pipeline

Every result is generated by the klbench runner (klbench run) and lands as a JSON file in results/<foundation>/<hardware>/ that must validate against schema/result-v3.schema.json. CI runs tools/validate.py on every pull request — invalid data cannot land — and the leaderboard rebuilds on merge.