ModelBenchkuhn-labs

Kuhn Labs / ModelProbe

Model scorecards

Quality and throughput, matched to hardware and serving settings.

How we measure ↓

768 output tokens per quality request · temperature 0 · 128 tokens for throughput. Same questions and budget across these configurations.

Selected: LFM2.5 24B-A2B Q4_K_M · original template

Configurations

4 measured
Model / configurationHardwareQuality¹1 request²8 concurrent²
4/6 checks · 1 runCPUIntel Xeon Gold 625456.3%29.533.1
3/6 checks · 1 run · partial · experimentCPUIntel Xeon Gold 6254Incomplete——
4/6 checks · 1 run · experimentCPUIntel Xeon Gold 625437.5%27.729.2
6/6 checks · 2 runsGPU4 × NVIDIA RTX 3090 24 GiB93.8%39.3–39.7291.4–291.5

¹ Equal-weight English + Mandarin sample. ² Output tokens/s; eight-request figures are aggregate throughput.

Selected configuration

LFM2.5 24B-A2B Q4_K_M

original template

CPU

Hardware & serving

Intel Xeon Gold 6254 · 16 vCPU · ~31 GiB usable RAM

Shared worker · warm · direct worker API

  • GGUF Q4_K_M
  • Ollama 0.40.3 / llama.cpp
  • context 131,072
  • KV cache f16
  • parallel requests 1
  • batch 2,048
  • flash attention enabled
  • queue 512
  • max loaded models 3
  • keep-alive -1
  • loaded model size ~18.08 GB
  • original Jinja template with an open thinking prefix
  • warm shared worker · direct API
  • quality output limit 768 tokens · throughput decode 128 tokens · temperature 0

Intelligence · quality sample

English6 / 8All questions returned
Mandarin3 / 8All questions returned
Local diagnostics4 / 11All questions returned

14 truncated · 0 parse errors · 0 request errors

Serving checks

4 / 6 passed per run.

Failed: Known-answer math, Reasoning separation.

Performance

One request
29.5 tok/s
Eight concurrent
33.1 aggregate tok/s
Measured duration
14m41s
Run limit
15 minutes
Samples
1 run
Sources & reproducibility

Measured 2026-11-06 – 2026-11-06 UTC. Checkpoint provenance: measured.

Model card ↗

Runner revision

d04818f6eeb8d994135f89a1a50854e07a118dc1

Raw result SHA-256

3174813617d68fa45723c8a90704a524fbc5cf0b9deda7564798b110802546a1

Scorecards

Intelligence · quality sample

50% English + 50% Mandarin · percent correct

LFM2.5 24B-A2B Q4_K_M · original template56.3%
LFM2.5 24B-A2B Q4_K_M · native thinkingIncomplete
LFM2.5 24B-A2B Q4_K_M · closed thinking prefix37.5%
Muse Glimmer 30B INT493.8%

Performance · output throughput

Eight concurrent requests · aggregate output tokens/s

LFM2.5 24B-A2B Q4_K_M · original template33.1
LFM2.5 24B-A2B Q4_K_M · native thinking—
LFM2.5 24B-A2B Q4_K_M · closed thinking prefix29.2
Muse Glimmer 30B INT4291.4–291.5
Bars show the minimum; labels show the observed range.

A quick barometer, with limits

ModelProbe runs six serving checks, 11 local diagnostics, eight English questions and eight Mandarin questions. The headline quality score gives each language 50% weight; local diagnostics stay separate. Repeats do not add unique questions.

This small sample is not a general intelligence ranking. English and Mandarin describe the test languages, not model origin or national leaderboards. Partial runs receive no headline score. Different output budgets stay in separate views.

Performance uses short, warm output sweeps at one and eight concurrent requests. It measures observed output throughput, not time to first token, sustained production load, or energy efficiency. Hardware and sharing conditions are shown with every configuration.

Charts use OpenUI with reviewed result data and fixed components. New measurements appear after publication. Reload published results ↻