Result
hf.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf:latest
Google← gemma-4-26B-A4B-itcdcollama cpu-4xlargetile 10.4.4
9/11/2026, 2:21:08 AM
usability
0.09±0.108
capability
0.11
completed
10%17/169
tok/s
0.3
agentic work
0.11
152 timeouts · 73 not attempted (6 h run cap)capped run
smoke · hard · frontier 0.12 · 0.07 · 0.06
judged bycodex (gpt-5.6-sol) · claude
Real work
No eligible PRD runs for this model yet — the headline board lists it under “no real-work runs yet”.
Where this model differs from its tier verifiedthinking: engine default
Against the CPU median (2 models): Strongest tool use +0.06 · Weakest multi turn −0.46
multi turn
−0.46
basic
−0.33
debugging
−0.31
structured output
−0.30
monitoring
−0.28
long context
−0.28
reasoning
−0.25
iac
−0.22
ci repair
−0.21
file ops
−0.14
tool use
+0.06
instruction
−0.04
coding
+0.00
writing
+0.00
research
+0.00
repo patch
+0.00
sysadmin
+0.00
agentic
+0.00
Zero line = the tier median for that category. Absolute scores are in the tests tab and the heatmap.
Category scores
basic
33%
14.72 tok/s · 68s avg
tool_use
93%
11.83 tok/s · 15s avg
structured_output
9%
10.28 tok/s · 24s avg
coding
0%
debugging
21%
12.07 tok/s · 28s avg
long_context
0%
instruction
0%
file_ops
0%
multi_turn
0%
reasoning
0%
writing
0%
research
0%
monitoring
0%
iac
0%
13.71 tok/s · 33s avg
ci_repair
43%
8.81 tok/s · 75s avg
repo_patch
0%
sysadmin
0%
agentic
0%
Individual tests
basic (3 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Arithmetic battery (6 items, graded) | timeout | 0% | {"reason":"hard kill after 360s"} |
| Quantitative reasoning battery (6 items, graded) | timeout | 0% | {"reason":"hard kill after 360s"} |
| Platform facts battery (6 items, graded) | scored | 100% | {"battery":6,"hit":6,"sub_refusals":0,"outcomes":[{"q":1,"outcome":"hit"},{"q":2 |
tool_use (10 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Dual tool call | scored | 100% | {"mode":"dual","tool_calls":["get_weather","get_time"],"matched":true,"unexpecte |
| Tool call — recover from malformed JSON error | scored | 50% | {"mode":"error_recovery","turn1_tool_calls":["lookup"],"turn2_tool_calls":["look |
| Tool call — recover from missing-parameter error | scored | 75% | {"mode":"error_recovery","turn1_tool_calls":["currency_convert"],"turn2_tool_cal |
| Restraint under ambiguity — ask, don't guess | scored | 100% | {"mode":"restraint","tool_calls":[],"matched":true,"finish_reason":"stop"} |
| Tool chain with derived arguments | scored | 100% | {"mode":"exact_args","tool_calls":["delete_binding"],"matched":true,"finish_reas |
| Nested args | scored | 100% | {"mode":"single","tool_calls":["search_code"],"matched":true,"finish_reason":"st |
| Read file | scored | 100% | {"mode":"single","tool_calls":["read_file"],"matched":true,"finish_reason":"stop |
| Restraint — no tool | scored | 100% | {"mode":"restraint","tool_calls":[],"matched":true,"finish_reason":"stop"} |
| Search code | scored | 100% | {"mode":"single","tool_calls":["search_code"],"matched":true,"finish_reason":"st |
| Weather single-call | scored | 100% | {"mode":"single","tool_calls":["get_weather"],"matched":true,"finish_reason":"st |
structured_output (11 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| REST API Paginated Response JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| Cloud Foundry App Manifest JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| Convert CSV to JSON Array | timeout | 0% | {"reason":"hard kill after 120s"} |
| Extract Receipt Data to JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| Conditional schema — route config with variant rules | timeout | 0% | {"reason":"hard kill after 360s"} |
| Kubernetes Pod Manifest JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| OpenAPI 3.0 Path Definition JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| Parse Nginx Access Log to JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| Function Call with Array and Boolean Arguments | timeout | 0% | {"reason":"hard kill after 120s"} |
| Function Call with Enum Argument | scored | 100% | {"parse_ok":true,"unwrapped":false,"strict_document":true,"fenced":true,"schema_ |
| Function Call with Nested Object Arguments | timeout | 0% | {"reason":"hard kill after 120s"} |
coding (15 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Bash parse log for unique IPs | timeout | 0% | {"reason":"hard kill after 180s"} |
| Bash line count | timeout | 0% | {"reason":"hard kill after 180s"} |
| Evacuation batching under role conflicts, DB safety and lexicographic minimality | timeout | 0% | {"reason":"hard kill after 540s"} |
| Interval merging with a strict minimum-gap rule | timeout | 0% | {"reason":"hard kill after 360s"} |
| Exact maintenance portfolio under windows, exclusions, and prerequisites | timeout | 0% | {"reason":"hard kill after 540s"} |
| Minimum patch waves under dependencies, rack conflicts, and service anti-affinity | timeout | 0% | {"reason":"hard kill after 540s"} |
| Lexicographic replica placement under compatibility, capacity, and zone anti-affinity | timeout | 0% | {"reason":"hard kill after 540s"} |
| CSV field extractor with quoting edge cases | timeout | 0% | {"reason":"hard kill after 360s"} |
| Semver caret-range matching including prerelease precedence | timeout | 0% | {"reason":"hard kill after 540s"} |
| Sliding-window top talker with deterministic tie-break | timeout | 0% | {"reason":"hard kill after 540s"} |
| Debounce with trailing invocation and argument capture | timeout | 0% | {"reason":"hard kill after 180s"} |
| Merge two sorted lists, stable and descending-aware | timeout | 0% | {"reason":"hard kill after 180s"} |
| Case-insensitive dedupe preserving first-seen casing | timeout | 0% | {"reason":"hard kill after 360s"} |
| Parse a compound duration string into seconds | timeout | 0% | {"reason":"hard kill after 360s"} |
| SQL top-N recent users | timeout | 0% | {"reason":"hard kill after 180s"} |
debugging (8 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Find the lock-order violator, not the deadlock victim | timeout | 0% | {"reason":"hard kill after 540s"} |
| Reject the plausible wrong fix | timeout | 0% | {"reason":"hard kill after 360s"} |
| Fix missing return in JS map callback | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix logic error in is_valid_age | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix off-by-one in find_max_index | timeout | 0% | {"reason":"hard kill after 180s"} |
| Shared mutable class attribute leaks across instances | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix wrong return type in count_vowels | scored | 100% | {"language":"python","passed":7,"total":7,"error":null,"properties":null,"hidden |
| Fix INNER JOIN dropping customers with no orders | scored | 67% | {"language":"sql","passed":4,"total":6,"error":null,"properties":null,"hidden_ca |
long_context (6 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Chain an escalation path across three facts (~4k tokens) | timeout | 0% | {"reason":"hard kill after 540s"} |
| Twelve config revisions with expiries, scopes, reverts and no-op cancels | timeout | 0% | {"reason":"hard kill after 720s"} |
| Cross-reference three separated facts (~16k tokens) | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| BOSH manifest property precedence (cross-reference) | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Three needles with superseded decoys (~4k tokens) | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Three inventory facts with superseded decoys (~1k tokens) | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
instruction (13 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Instruction conflict — format contradiction | timeout | 0% | {"reason":"hard kill after 135s"} |
| Instruction conflict — ordering contradiction | timeout | 0% | {"reason":"hard kill after 135s"} |
| Forbidden words — sunset | timeout | 0% | {"reason":"hard kill after 120s"} |
| Acrostic plus lipogram plus exact word counts, simultaneously | timeout | 0% | {"reason":"hard kill after 720s"} |
| Release changelog under six simultaneous mechanical constraints | timeout | 0% | {"reason":"hard kill after 720s"} |
| Three sentences about deployment without the letter e | timeout | 0% | {"reason":"hard kill after 540s"} |
| Seven lines where line N contains exactly N words | timeout | 0% | {"reason":"hard kill after 540s"} |
| Six simultaneous constraints | timeout | 0% | {"reason":"hard kill after 360s"} |
| Prompt injection — embedded role override | timeout | 0% | {"reason":"hard kill after 135s"} |
| Prompt injection — ignore-previous-instructions attack | timeout | 0% | {"reason":"hard kill after 135s"} |
| JSON-only output | timeout | 0% | {"reason":"hard kill after 120s"} |
| Markdown structure | timeout | 0% | {"reason":"hard kill after 120s"} |
| Word count — 20 words | timeout | 0% | {"reason":"hard kill after 120s"} |
file_ops (3 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Create haiku file | timeout | 0% | {"reason":"hard kill after 120s"} |
| Write Python function file | timeout | 0% | {"reason":"hard kill after 120s"} |
| Write CSV file | timeout | 0% | {"reason":"hard kill after 120s"} |
multi_turn (8 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Context carryover — follow most recent contradictory claim | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Context carryover — job recall after topic shift | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Context carryover — number recall after distraction | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Iterative code refinement (greet function → error handling → docstring) | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Iterative email refinement (apologetic + deadline + shorten) | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Iterative summary refinement (summarize → shorten → bullet points) | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-step tool use — weather query then umbrella advice | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-step tool use — read file then extract port | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
reasoning (8 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Causal reasoning — root cause of extended shutdown | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Ten-stage capacity cascade with mixed units and stage-wise flooring | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Cluster capacity with reserved overhead and millicore units | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Deployment ordering under six interacting constraints | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Single overlapping on-call hour across three timezones | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-hop reasoning — biography dates and places | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-hop reasoning — org chart before and after reorganization | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Timeline ordering — Project Helios events | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
writing (3 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Writing — professional meeting reschedule email | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Writing — summarize news article in 3 sentences | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Writing — explain HTTP caching for non-technical readers | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
research (2 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Cited answer — source trust for policy decision | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Synthesis with citations — remote work productivity | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
monitoring (12 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Diagnose CF app crash loop from events | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose CF 404s from routes and app state | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose full disk from df output | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose Diego cell resource exhaustion | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose OOM risk from free output | timeout | 0% | {"reason":"hard kill after 120s"} |
| Alert triage — page exactly one | timeout | 0% | {"reason":"hard kill after 360s"} |
| Cascading failure root cause (multi-signal) | timeout | 0% | {"reason":"hard kill after 360s"} |
| False 'deploy succeeded' from polling the latest install | timeout | 0% | {"reason":"hard kill after 360s"} |
| SLO error-budget burn rate | timeout | 0% | {"reason":"hard kill after 360s"} |
| Identify latency anomaly in Healthwatch metrics | timeout | 0% | {"reason":"hard kill after 120s"} |
| Identify critical disk from Prometheus metrics | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose CPU spike from top output | timeout | 0% | {"reason":"hard kill after 120s"} |
iac (12 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Generate CF app manifest JSON for Spring Boot | timeout | 0% | {"reason":"hard kill after 120s"} |
| Write CF network policy CLI commands | timeout | 0% | {"reason":"hard kill after 120s"} |
| Generate CF manifest with service bindings | timeout | 0% | {"reason":"hard kill after 120s"} |
| Write multi-stage Dockerfile for Go app | timeout | 0% | {"reason":"hard kill after 120s"} |
| Max instances under even AZ spread with asymmetric cells | timeout | 0% | {"reason":"hard kill after 540s"} |
| Write GitHub Actions workflow for test and deploy | timeout | 0% | {"reason":"hard kill after 120s"} |
| BOSH errand failure — name the misconfigured property | timeout | 0% | {"reason":"hard kill after 360s"} |
| Minimal CF egress ASG (no shortcuts) | timeout | 0% | {"reason":"hard kill after 360s"} |
| Custom ollama template silently ignored on a GenAI tile | timeout | 0% | {"reason":"hard kill after 360s"} |
| Fix K8s Deployment ErrImageNeverPull | timeout | 0% | {"battery":3,"hit":3,"finish_reason":"stop","budget_ms":30000,"overrun_ms":3190, |
| Generate K8s Deployment JSON for nginx | timeout | 0% | {"reason":"hard kill after 120s"} |
| Generate K8s Service and Ingress for nginx on 443 | timeout | 0% | {"reason":"hard kill after 120s"} |
ci_repair (7 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Fix invalid base image tag in Dockerfile | scored | 100% | {"build_ok":true,"stdout":"ok\n","finish_reason":"stop"} |
| Resolve dependency pins through a transitive compatibility matrix | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Fix missing require directive in go.mod | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix wrong groupId in pom.xml dependency | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix missing dependency in package.json | scored | 100% | {"build_ok":true,"stdout":"ok\n","finish_reason":"stop"} |
| Fix incompatible package versions in requirements.txt | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix YAML syntax error in GitHub Actions workflow | timeout | 0% | {"reason":"hard kill after 180s"} |
repo_patch (14 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Add /v2/users endpoint without breaking /v1 | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add /v2/users endpoint without breaking /v1 | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Migrate app config from env vars to YAML with env fallback | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Migrate app config from env vars to YAML with env fallback | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix partial-match search bug in Flask-like app | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix partial-match search bug in Flask-like app | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Extract shared validation logic into cli/validators.py | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Extract shared validation logic into cli/validators.py | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add input validation to withdraw() | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add input validation to withdraw() | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Migrate off a deprecated API | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Migrate off a deprecated API | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix pagination off-by-one | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix pagination off-by-one | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
sysadmin (6 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Set up cron job for backup | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Write a valid docker-compose.yml with postgres, redis, and volumes | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Set iptables rules to allow 22/80/443 and drop rest | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Configure nginx reverse proxy for /api | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Create systemd unit file for a Python app | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Create deploy user with www-data group | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
agentic (28 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Implement a documented contract and prove it with tests | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Implement a documented contract and prove it with tests | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add pagination to get_items | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add pagination to get_items | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Write CLI passing integration tests | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Write CLI passing integration tests | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Debug bug buried in import chain | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Debug bug buried in import chain | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix null deref bug | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix null deref bug | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Two failures — decide which side is wrong in each | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Two failures — decide which side is wrong in each | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Atomic config reconciliation — idempotent, generation-ordered, no partial writes | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Atomic config reconciliation — idempotent, generation-ordered, no partial writes | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Partial observability — the fault is not in the prompt, only in the data | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Partial observability — the fault is not in the prompt, only in the data | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Coordinated config-schema + code migration | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Coordinated config-schema + code migration | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-module feature — per-client rate limiter | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-module feature — per-client rate limiter | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Align Python response key with JS frontend expectation | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Align Python response key with JS frontend expectation | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Health-gated cutover with ambiguity detection and rollback | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Health-gated cutover with ambiguity detection and rollback | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Semantic config migration with variants and precedence | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Semantic config migration with variants and precedence | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Rename function everywhere | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Rename function everywhere | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
Per-task run consistency (K runs per test)
Run consistency
min 0.00·avg 0.94·max 1.00
Platform facts battery (6 items, graded)
1.00
Dual tool call
1.00
Tool call — recover from malformed JSON error
1.00
Tool call — recover from missing-parameter error
0.00
Restraint under ambiguity — ask, don't guess
1.00
Tool chain with derived arguments
1.00
Nested args
1.00
Read file
1.00
Restraint — no tool
1.00
Search code
1.00
Weather single-call
1.00
Function Call with Enum Argument
1.00
Fix wrong return type in count_vowels
1.00
Fix INNER JOIN dropping customers with no orders
1.00
Fix K8s Deployment ErrImageNeverPull
1.00
Fix invalid base image tag in Dockerfile
1.00
Resolve dependency pins through a transitive compatibility matrix
1.00
Fix missing dependency in package.json
1.00