ModelBenchkuhn-labs

Historical benchmark · different suite. View current ModelProbe scorecards →

Current capability suite

All models

Current published results in a single stream, badged by hardware comparability tier. Tiers come from the exact recorded GPU model. GPU tiers and the hosted subscription reference headline the capability composite; CPU and entry GPU headline timeout-honest usability — open a tier board for rank-comparable numbers. Retired era data remains available only in the source archive.

How to read this board
  • The number. Each row shows its tier’s headline metric: capability composite for GPU rows, timeout-honest usability for CPU and entry-GPU rows.
  • Ships. Ships = mean of per-PRD mean scores over the non-saturated, mechanically graded PRDs, run under the opencode on-prem harness on placement-attested workers. Means, not medians: PRD scores are coarse and bimodal, so a median can hide a 40% failure rate behind a 1.0. Saturated PRDs (every model mean ≥ 0.95) are excluded, not hidden. Provisional until every model × PRD cell has at least two runs in one era. It is the site's headline; this board is the diagnostic behind it — on this panel the capability composite correlates negatively (ρ ≈ −0.6, n = 6) with real-work outcome; it is a diagnostic, not a ranking.
  • Shared ranks. Rows whose gap to the tier anchor is below the paired minimum detectable effect (power 0.80) are statistically indistinguishable at this suite size, so they share a rank and a T chip instead of being ordered by noise.
  • The chips. ⚠ capped — the run hit its wall-clock cap; not attempted: the whole run hit its wall-clock cap before these items were sent to the model. They still score 0 (runner rule) but are not a verdict on the model. partial — some categories produced no gradable rows, so the composite is not comparable and the row is excluded from statistical tiering. ◆ verified — the serving worker for this row was verified against BOSH logs.
GPU · 5 →Entry GPU (16 GB class) · 1 →CPU · 3 →
1
Qwen/Qwen3.8-27B-FP8GPU verified
Alibaba Cloud
fp8
0.95±0.027
capability
ndc · vllm98% done · 4 timeouts67.5 tok/s
Categories:
Sep 11
2
google/gemma-4-31B-it-qat-w4a16-ctGPU verifiedno-think
Google
qat-w4a16
0.92±0.026
capability
ndc · vllm100% done47.3 tok/s
Categories:
Sep 10
3
openai/gpt-oss-120bGPU verified
OpenAI
0.92±0.045
capability
ndc · vllm99% done · 1 timeout79.2 tok/s
Categories:
Sep 10
4
deepseek-ai/DeepSeek-V4-Flash-0731GPU verifiedno-think
DeepSeek
0.89±0.048
capability
ndc · vllm100% done57.4 tok/s
Categories:
Sep 10
5
poolside/Laguna-S-2.1-NVFP4GPU verifiedno-think
poolside
0.87±0.050
capability
ndc · vllm100% done98.6 tok/s
Categories:
Sep 10
6
hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_MCPU⚠
Alibaba Cloud
q4
0.67±0.110
usability
cdc · ollama82% done · 28 timeouts · 10 not attempted (6 h run cap)14.3 tok/s
Categories:
Aug 31
7
hf.co/google/gemma-4-12B-it-qat-q4_0-gguf:latestENTRY GPU⚠ verified
Google
q4
0.48±0.166
usability
cdc · ollama54% done · 78 timeouts · 36 not attempted (6 h run cap)17.0 tok/s
Categories:
Sep 10
8
hf.co/unsloth/gpt-oss-20b-GGUF:UD-Q4_K_XLCPU⚠ verified
OpenAI
q4
0.35±0.172
usability
cdc · ollama38% done · 104 timeouts · 54 not attempted (6 h run cap)11.5 tok/s
Categories:
Sep 10
9
hf.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf:latestCPU⚠ verified
Google
q4
0.09±0.108
usability
cdc · ollama10% done · 152 timeouts · 73 not attempted (6 h run cap)11.4 tok/s
Categories:
Sep 11

One stream, badged by tier. Each row shows its tier's headline metric, so cross-tier order is context, not a verdict — the tier boards above carry the rank-comparable MDE tiers.