Result
hf.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf:latest
Google← gemma-4-26B-A4B-itcdcollama cpu-4xlargetile 10.4.4
8/30/2026, 9:46:41 PM
usability
0.24±0.156
capability
0.27
completed
25%42/165
tok/s
1.4
123 timeouts · 59 not attempted (6 h run cap)capped run
smoke · hard · frontier 0.28 · 0.20 · 0.17
Category scores
basic
33%
18.17 tok/s · 52s avg
tool_use
93%
10.14 tok/s · 11s avg
structured_output
52%
17.98 tok/s · 24s avg
coding
0%
debugging
25%
12.27 tok/s · 28s avg
long_context
89%
12.2 tok/s · 56s avg
instruction
0%
file_ops
0%
multi_turn
59%
5.1 tok/s · 52s avg
reasoning
0%
writing
0%
research
0%
monitoring
25%
15.7 tok/s · 39s avg
iac
21%
13.89 tok/s · 76s avg
ci_repair
86%
15.07 tok/s · 47s avg
repo_patch
0%
sysadmin
0%
agentic
0%
Individual tests
basic (3 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Arithmetic battery (6 items, graded) | timeout | 0% | {"reason":"hard kill after 360s"} |
| Quantitative reasoning battery (6 items, graded) | timeout | 0% | {"reason":"hard kill after 360s"} |
| Platform facts battery (6 items, graded) | scored | 100% | {"battery":6,"hit":6,"sub_refusals":0,"outcomes":[{"q":1,"outcome":"hit"},{"q":2 |
tool_use (10 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Dual tool call | scored | 100% | {"mode":"dual","tool_calls":["get_weather","get_time"],"matched":true,"unexpecte |
| Tool call — recover from malformed JSON error | scored | 50% | {"mode":"error_recovery","turn1_tool_calls":["lookup"],"turn2_tool_calls":["look |
| Tool call — recover from missing-parameter error | scored | 75% | {"mode":"error_recovery","turn1_tool_calls":["currency_convert"],"turn2_tool_cal |
| Restraint under ambiguity — ask, don't guess | scored | 100% | {"mode":"restraint","tool_calls":[],"matched":true,"finish_reason":"stop"} |
| Tool chain with derived arguments | scored | 100% | {"mode":"exact_args","tool_calls":["delete_binding"],"matched":true,"finish_reas |
| Nested args | scored | 100% | {"mode":"single","tool_calls":["search_code"],"matched":true,"finish_reason":"st |
| Read file | scored | 100% | {"mode":"single","tool_calls":["read_file"],"matched":true,"finish_reason":"stop |
| Restraint — no tool | scored | 100% | {"mode":"restraint","tool_calls":[],"matched":true,"finish_reason":"stop"} |
| Search code | scored | 100% | {"mode":"single","tool_calls":["search_code"],"matched":true,"finish_reason":"st |
| Weather single-call | scored | 100% | {"mode":"single","tool_calls":["get_weather"],"matched":true,"finish_reason":"st |
structured_output (11 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| REST API Paginated Response JSON | scored | 100% | {"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_ |
| Cloud Foundry App Manifest JSON | scored | 100% | {"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_ |
| Convert CSV to JSON Array | timeout | 0% | {"parse_ok":true,"strict_document":true,"fenced":false,"schema_score":1,"content |
| Extract Receipt Data to JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| Conditional schema — route config with variant rules | scored | 70% | {"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_ |
| Kubernetes Pod Manifest JSON | scored | 100% | {"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_ |
| OpenAPI 3.0 Path Definition JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| Parse Nginx Access Log to JSON | timeout | 0% | {"reason":"hard kill after 120s"} |
| Function Call with Array and Boolean Arguments | scored | 100% | {"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_ |
| Function Call with Enum Argument | scored | 100% | {"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_ |
| Function Call with Nested Object Arguments | timeout | 0% | {"reason":"hard kill after 120s"} |
coding (15 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Bash parse log for unique IPs | timeout | 0% | {"reason":"hard kill after 180s"} |
| Bash line count | timeout | 0% | {"reason":"hard kill after 180s"} |
| Evacuation batching under role conflicts, DB safety and lexicographic minimality | timeout | 0% | {"reason":"hard kill after 540s"} |
| Interval merging with a strict minimum-gap rule | timeout | 0% | {"reason":"hard kill after 360s"} |
| Exact maintenance portfolio under windows, exclusions, and prerequisites | timeout | 0% | {"reason":"hard kill after 540s"} |
| Minimum patch waves under dependencies, rack conflicts, and service anti-affinity | timeout | 0% | {"reason":"hard kill after 540s"} |
| Lexicographic replica placement under compatibility, capacity, and zone anti-affinity | timeout | 0% | {"reason":"hard kill after 540s"} |
| CSV field extractor with quoting edge cases | timeout | 0% | {"reason":"hard kill after 360s"} |
| Semver caret-range matching including prerelease precedence | timeout | 0% | {"reason":"hard kill after 540s"} |
| Sliding-window top talker with deterministic tie-break | timeout | 0% | {"reason":"hard kill after 540s"} |
| Debounce with trailing invocation and argument capture | timeout | 0% | {"reason":"hard kill after 180s"} |
| Merge two sorted lists, stable and descending-aware | timeout | 0% | {"reason":"hard kill after 180s"} |
| Case-insensitive dedupe preserving first-seen casing | timeout | 0% | {"reason":"hard kill after 360s"} |
| Parse a compound duration string into seconds | timeout | 0% | {"reason":"hard kill after 360s"} |
| SQL top-N recent users | timeout | 0% | {"reason":"hard kill after 180s"} |
debugging (8 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Find the lock-order violator, not the deadlock victim | timeout | 0% | {"reason":"hard kill after 540s"} |
| Reject the plausible wrong fix | timeout | 0% | {"reason":"hard kill after 360s"} |
| Fix missing return in JS map callback | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix logic error in is_valid_age | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix off-by-one in find_max_index | timeout | 0% | {"reason":"hard kill after 180s"} |
| Shared mutable class attribute leaks across instances | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix wrong return type in count_vowels | scored | 100% | {"language":"python","passed":7,"total":7,"error":null,"properties":null,"hidden |
| Fix INNER JOIN dropping customers with no orders | scored | 100% | {"language":"sql","passed":4,"total":4,"error":null,"properties":null,"hidden_ca |
long_context (6 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Chain an escalation path across three facts (~4k tokens) | scored | 33% | {"battery":3,"hit":1,"finish_reason":"stop"} |
| Twelve config revisions with expiries, scopes, reverts and no-op cancels | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Cross-reference three separated facts (~16k tokens) | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| BOSH manifest property precedence (cross-reference) | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Three needles with superseded decoys (~4k tokens) | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Three inventory facts with superseded decoys (~1k tokens) | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
instruction (13 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Instruction conflict — format contradiction | timeout | 0% | {"reason":"hard kill after 135s"} |
| Instruction conflict — ordering contradiction | timeout | 0% | {"reason":"hard kill after 135s"} |
| Forbidden words — sunset | timeout | 0% | {"reason":"hard kill after 120s"} |
| Acrostic plus lipogram plus exact word counts, simultaneously | timeout | 0% | {"reason":"hard kill after 720s"} |
| Release changelog under six simultaneous mechanical constraints | timeout | 0% | {"reason":"hard kill after 720s"} |
| Three sentences about deployment without the letter e | timeout | 0% | {"reason":"hard kill after 540s"} |
| Seven lines where line N contains exactly N words | timeout | 0% | {"reason":"hard kill after 540s"} |
| Six simultaneous constraints | timeout | 0% | {"reason":"hard kill after 360s"} |
| Prompt injection — embedded role override | timeout | 0% | {"reason":"hard kill after 135s"} |
| Prompt injection — ignore-previous-instructions attack | timeout | 0% | {"reason":"hard kill after 135s"} |
| JSON-only output | timeout | 0% | {"reason":"hard kill after 120s"} |
| Markdown structure | timeout | 0% | {"reason":"hard kill after 120s"} |
| Word count — 20 words | timeout | 0% | {"reason":"hard kill after 120s"} |
file_ops (3 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Create haiku file | timeout | 0% | {"reason":"hard kill after 120s"} |
| Write Python function file | timeout | 0% | {"reason":"hard kill after 120s"} |
| Write CSV file | timeout | 0% | {"reason":"hard kill after 120s"} |
multi_turn (8 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Context carryover — follow most recent contradictory claim | scored | 75% | {"turns":[{"user":"The capital of France is Paris.\n","response":"That is correc |
| Context carryover — job recall after topic shift | scored | 100% | {"turns":[{"user":"Alice works as a software engineer.\n","response":"Understood |
| Context carryover — number recall after distraction | scored | 100% | {"turns":[{"user":"Remember these numbers: 7, 13, 25.\n","response":"Understood. |
| Iterative code refinement (greet function → error handling → docstring) | scored | 100% | {"turns":[{"user":"Write a Python function called greet(name) that returns \"Hel |
| Iterative email refinement (apologetic + deadline + shorten) | timeout | 0% | {"turns":[{"user":"Write a short email to Sarah explaining that a project delive |
| Iterative summary refinement (summarize → shorten → bullet points) | scored | 100% | {"turns":[{"user":"Summarize this: \"The quick brown fox jumps over the lazy dog |
| Multi-step tool use — weather query then umbrella advice | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-step tool use — read file then extract port | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
reasoning (8 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Causal reasoning — root cause of extended shutdown | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Ten-stage capacity cascade with mixed units and stage-wise flooring | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Cluster capacity with reserved overhead and millicore units | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Deployment ordering under six interacting constraints | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Single overlapping on-call hour across three timezones | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-hop reasoning — biography dates and places | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-hop reasoning — org chart before and after reorganization | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Timeline ordering — Project Helios events | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
writing (3 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Writing — professional meeting reschedule email | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Writing — summarize news article in 3 sentences | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Writing — explain HTTP caching for non-technical readers | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
research (2 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Cited answer — source trust for policy decision | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Synthesis with citations — remote work productivity | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
monitoring (12 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Diagnose CF app crash loop from events | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose CF 404s from routes and app state | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose full disk from df output | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose Diego cell resource exhaustion | timeout | 0% | {"reason":"hard kill after 120s"} |
| Diagnose OOM risk from free output | timeout | 0% | {"reason":"hard kill after 120s"} |
| Alert triage — page exactly one | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Cascading failure root cause (multi-signal) | timeout | 0% | {"reason":"hard kill after 360s"} |
| False 'deploy succeeded' from polling the latest install | timeout | 0% | {"reason":"hard kill after 360s"} |
| SLO error-budget burn rate | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Identify latency anomaly in Healthwatch metrics | timeout | 0% | {"reason":"hard kill after 120s"} |
| Identify critical disk from Prometheus metrics | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Diagnose CPU spike from top output | timeout | 0% | {"battery":3,"hit":3,"finish_reason":"stop","budget_ms":30000,"overrun_ms":267," |
iac (12 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Generate CF app manifest JSON for Spring Boot | timeout | 0% | {"reason":"hard kill after 120s"} |
| Write CF network policy CLI commands | timeout | 0% | {"reason":"hard kill after 120s"} |
| Generate CF manifest with service bindings | timeout | 0% | {"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":0,"content_ |
| Write multi-stage Dockerfile for Go app | timeout | 0% | {"reason":"hard kill after 120s"} |
| Max instances under even AZ spread with asymmetric cells | scored | 100% | {"battery":2,"hit":2,"finish_reason":"stop"} |
| Write GitHub Actions workflow for test and deploy | timeout | 0% | {"reason":"hard kill after 120s"} |
| BOSH errand failure — name the misconfigured property | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Minimal CF egress ASG (no shortcuts) | scored | 50% | {"build_ok":false,"stderr":"Traceback (most recent call last):\n File \"<stdin> |
| Custom ollama template silently ignored on a GenAI tile | timeout | 0% | {"reason":"hard kill after 360s"} |
| Fix K8s Deployment ErrImageNeverPull | timeout | 0% | {"battery":3,"hit":3,"finish_reason":"stop","budget_ms":30000,"overrun_ms":5332, |
| Generate K8s Deployment JSON for nginx | timeout | 0% | {"reason":"hard kill after 120s"} |
| Generate K8s Service and Ingress for nginx on 443 | timeout | 0% | {"reason":"hard kill after 120s"} |
ci_repair (7 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Fix invalid base image tag in Dockerfile | scored | 100% | {"build_ok":true,"stdout":"ok\n","finish_reason":"stop"} |
| Resolve dependency pins through a transitive compatibility matrix | scored | 100% | {"battery":3,"hit":3,"finish_reason":"stop"} |
| Fix missing require directive in go.mod | scored | 100% | {"build_ok":true,"stdout":"ok\n","finish_reason":"stop"} |
| Fix wrong groupId in pom.xml dependency | scored | 100% | {"build_ok":true,"stdout":"ok\n","finish_reason":"stop"} |
| Fix missing dependency in package.json | scored | 100% | {"build_ok":true,"stdout":"ok\n","finish_reason":"stop"} |
| Fix incompatible package versions in requirements.txt | timeout | 0% | {"reason":"hard kill after 180s"} |
| Fix YAML syntax error in GitHub Actions workflow | scored | 100% | {"build_ok":true,"stdout":"ok\n","finish_reason":"stop"} |
repo_patch (14 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Add /v2/users endpoint without breaking /v1 | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add /v2/users endpoint without breaking /v1 | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Migrate app config from env vars to YAML with env fallback | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Migrate app config from env vars to YAML with env fallback | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix partial-match search bug in Flask-like app | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix partial-match search bug in Flask-like app | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Extract shared validation logic into cli/validators.py | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Extract shared validation logic into cli/validators.py | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add input validation to withdraw() | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add input validation to withdraw() | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Migrate off a deprecated API | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Migrate off a deprecated API | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix pagination off-by-one | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix pagination off-by-one | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
sysadmin (6 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Set up cron job for backup | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Write a valid docker-compose.yml with postgres, redis, and volumes | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Set iptables rules to allow 22/80/443 and drop rest | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Configure nginx reverse proxy for /api | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Create systemd unit file for a Python app | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Create deploy user with www-data group | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
agentic (24 tests)
| Test | Status | Score | Details |
|---|---|---|---|
| Implement a documented contract and prove it with tests | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Implement a documented contract and prove it with tests | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add pagination to get_items | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Add pagination to get_items | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Write CLI passing integration tests | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Write CLI passing integration tests | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Debug bug buried in import chain | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Debug bug buried in import chain | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix null deref bug | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Fix null deref bug | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Two failures — decide which side is wrong in each | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Two failures — decide which side is wrong in each | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Atomic config reconciliation — idempotent, generation-ordered, no partial writes | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Atomic config reconciliation — idempotent, generation-ordered, no partial writes | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Partial observability — the fault is not in the prompt, only in the data | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Partial observability — the fault is not in the prompt, only in the data | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Coordinated config-schema + code migration | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Coordinated config-schema + code migration | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-module feature — per-client rate limiter | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Multi-module feature — per-client rate limiter | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Align Python response key with JS frontend expectation | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Align Python response key with JS frontend expectation | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Rename function everywhere | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
| Rename function everywhere | timeout | 0% | {"reason":"run_time_cap_exceeded"} |
Per-task run consistency (K runs per test)
Run consistency
min 0.00·avg 0.93·max 1.00
Platform facts battery (6 items, graded)
1.00
Dual tool call
1.00
Tool call — recover from malformed JSON error
1.00
Tool call — recover from missing-parameter error
0.00
Restraint under ambiguity — ask, don't guess
1.00
Tool chain with derived arguments
1.00
Nested args
1.00
Read file
1.00
Restraint — no tool
1.00
Search code
1.00
Weather single-call
1.00
REST API Paginated Response JSON
1.00
Cloud Foundry App Manifest JSON
1.00
Convert CSV to JSON Array
1.00
Conditional schema — route config with variant rules
1.00
Kubernetes Pod Manifest JSON
1.00
Function Call with Array and Boolean Arguments
1.00
Function Call with Enum Argument
1.00
Fix wrong return type in count_vowels
1.00
Fix INNER JOIN dropping customers with no orders
1.00
Alert triage — page exactly one
1.00
SLO error-budget burn rate
1.00
Identify critical disk from Prometheus metrics
1.00
Diagnose CPU spike from top output
1.00
Generate CF manifest with service bindings
1.00
Max instances under even AZ spread with asymmetric cells
1.00
BOSH errand failure — name the misconfigured property
1.00
Minimal CF egress ASG (no shortcuts)
1.00
Fix K8s Deployment ErrImageNeverPull
1.00
Fix invalid base image tag in Dockerfile
1.00
Resolve dependency pins through a transitive compatibility matrix
1.00
Fix missing require directive in go.mod
1.00
Fix wrong groupId in pom.xml dependency
1.00
Fix missing dependency in package.json
1.00
Fix YAML syntax error in GitHub Actions workflow
1.00
Chain an escalation path across three facts (~4k tokens)
0.00
Twelve config revisions with expiries, scopes, reverts and no-op cancels
1.00
Cross-reference three separated facts (~16k tokens)
1.00
BOSH manifest property precedence (cross-reference)
1.00
Three needles with superseded decoys (~4k tokens)
1.00
Three inventory facts with superseded decoys (~1k tokens)
1.00
Context carryover — follow most recent contradictory claim
1.00
Context carryover — job recall after topic shift
0.23
Context carryover — number recall after distraction
1.00
Iterative code refinement (greet function → error handling → docstring)
1.00
Iterative email refinement (apologetic + deadline + shorten)
0.67
Iterative summary refinement (summarize → shorten → bullet points)
1.00