Result

hf.co/google/gemma-4-12B-it-qat-q4_0-gguf:latest

Google← gemma-4-12B-it

cdcollama t4-largetile 10.4.4

8/30/2026, 6:10:12 PM

usability

0.53±0.129

capability

0.66

completed

60%99/165

tok/s

4.1

66 timeouts · 27 not attempted (6 h run cap)capped run

smoke · hard · frontier 0.62 · 0.43 · 0.39

Category scores

basic

100%

21.21 tok/s · 81s avg

tool_use

88%

14.68 tok/s · 7s avg

structured_output

61%

19.7 tok/s · 20s avg

coding

20%

18.3 tok/s · 29s avg

debugging

75%

20.84 tok/s · 28s avg

long_context

89%

18.37 tok/s · 29s avg

instruction

62%

20.38 tok/s · 66s avg

file_ops

62%

19.02 tok/s · 18s avg

multi_turn

93%

7.87 tok/s · 25s avg

reasoning

63%

13.26 tok/s · 52s avg

writing

89%

11.99 tok/s · 70s avg

research

98%

11.4 tok/s · 79s avg

monitoring

65%

20.89 tok/s · 42s avg

iac

29%

19.48 tok/s · 44s avg

ci_repair

86%

19.21 tok/s · 28s avg

repo_patch

53%

sysadmin

50%

21.81 tok/s · 47s avg

agentic

Individual tests

basic (3 tests)

Test	Status	Score	Details
Arithmetic battery (6 items, graded)	scored	100%	{"battery":6,"hit":6,"sub_refusals":0,"outcomes":[{"q":1,"outcome":"hit"},{"q":2
Quantitative reasoning battery (6 items, graded)	scored	100%	{"battery":6,"hit":6,"sub_refusals":0,"outcomes":[{"q":1,"outcome":"hit"},{"q":2
Platform facts battery (6 items, graded)	scored	100%	{"battery":6,"hit":6,"sub_refusals":0,"outcomes":[{"q":1,"outcome":"hit"},{"q":2

tool_use (10 tests)

Test	Status	Score	Details
Dual tool call	scored	50%	{"mode":"dual","tool_calls":["get_weather"],"matched":false,"unexpected":[],"fin
Tool call — recover from malformed JSON error	scored	50%	{"mode":"error_recovery","turn1_tool_calls":["lookup"],"turn2_tool_calls":["look
Tool call — recover from missing-parameter error	scored	75%	{"mode":"error_recovery","turn1_tool_calls":["currency_convert"],"turn2_tool_cal
Restraint under ambiguity — ask, don't guess	scored	100%	{"mode":"restraint","tool_calls":[],"matched":true,"finish_reason":"stop"}
Tool chain with derived arguments	scored	100%	{"mode":"exact_args","tool_calls":["delete_binding"],"matched":true,"finish_reas
Nested args	scored	100%	{"mode":"single","tool_calls":["search_code"],"matched":true,"finish_reason":"st
Read file	scored	100%	{"mode":"single","tool_calls":["read_file"],"matched":true,"finish_reason":"stop
Restraint — no tool	scored	100%	{"mode":"restraint","tool_calls":[],"matched":true,"finish_reason":"stop"}
Search code	scored	100%	{"mode":"single","tool_calls":["search_code"],"matched":true,"finish_reason":"st
Weather single-call	scored	100%	{"mode":"single","tool_calls":["get_weather"],"matched":true,"finish_reason":"st

structured_output (11 tests)

Test	Status	Score	Details
REST API Paginated Response JSON	scored	100%	{"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_
Cloud Foundry App Manifest JSON	scored	100%	{"parse_ok":true,"strict_document":true,"fenced":false,"schema_score":1,"content
Convert CSV to JSON Array	scored	100%	{"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_
Extract Receipt Data to JSON	timeout	0%	{"reason":"hard kill after 120s"}
Conditional schema — route config with variant rules	scored	70%	{"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_
Kubernetes Pod Manifest JSON	scored	100%	{"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_
OpenAPI 3.0 Path Definition JSON	timeout	0%	{"reason":"hard kill after 120s"}
Parse Nginx Access Log to JSON	timeout	0%	{"reason":"hard kill after 120s"}
Function Call with Array and Boolean Arguments	scored	100%	{"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_
Function Call with Enum Argument	scored	100%	{"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":1,"content_
Function Call with Nested Object Arguments	timeout	0%	{"reason":"hard kill after 120s"}

coding (15 tests)

Test	Status	Score	Details
Bash parse log for unique IPs	scored	100%	{"language":"bash","passed":7,"total":7,"error":null,"properties":null,"hidden_c
Bash line count	timeout	0%	{"reason":"hard kill after 180s"}
Evacuation batching under role conflicts, DB safety and lexicographic minimality	timeout	0%	{"reason":"hard kill after 540s"}
Interval merging with a strict minimum-gap rule	timeout	0%	{"reason":"hard kill after 360s"}
Exact maintenance portfolio under windows, exclusions, and prerequisites	timeout	0%	{"reason":"hard kill after 540s"}
Minimum patch waves under dependencies, rack conflicts, and service anti-affinity	timeout	0%	{"reason":"hard kill after 540s"}
Lexicographic replica placement under compatibility, capacity, and zone anti-affinity	timeout	0%	{"reason":"hard kill after 540s"}
CSV field extractor with quoting edge cases	timeout	0%	{"reason":"hard kill after 360s"}
Semver caret-range matching including prerelease precedence	timeout	0%	{"reason":"hard kill after 540s"}
Sliding-window top talker with deterministic tie-break	timeout	0%	{"reason":"hard kill after 540s"}
Debounce with trailing invocation and argument capture	timeout	0%	{"reason":"hard kill after 180s"}
Merge two sorted lists, stable and descending-aware	timeout	0%	{"reason":"hard kill after 180s"}
Case-insensitive dedupe preserving first-seen casing	scored	100%	{"language":"python","passed":6,"total":6,"error":null,"properties":null,"hidden
Parse a compound duration string into seconds	timeout	0%	{"reason":"hard kill after 360s"}
SQL top-N recent users	scored	100%	{"language":"sql","passed":4,"total":4,"error":null,"properties":null,"hidden_ca

debugging (8 tests)

Test	Status	Score	Details
Find the lock-order violator, not the deadlock victim	scored	100%	{"battery":2,"hit":2,"finish_reason":"stop"}
Reject the plausible wrong fix	scored	100%	{"language":"python","passed":3,"total":3,"error":null,"properties":null,"hidden
Fix missing return in JS map callback	scored	100%	{"language":"javascript","passed":3,"total":3,"error":null,"properties":null,"hi
Fix logic error in is_valid_age	scored	100%	{"language":"python","passed":7,"total":7,"error":null,"properties":null,"hidden
Fix off-by-one in find_max_index	scored	100%	{"language":"python","passed":7,"total":7,"error":null,"properties":null,"hidden
Shared mutable class attribute leaks across instances	timeout	0%	{"reason":"hard kill after 180s"}
Fix wrong return type in count_vowels	scored	100%	{"language":"python","passed":7,"total":7,"error":null,"properties":null,"hidden
Fix INNER JOIN dropping customers with no orders	timeout	0%	{"reason":"hard kill after 180s"}

long_context (6 tests)

Test	Status	Score	Details
Chain an escalation path across three facts (~4k tokens)	scored	33%	{"battery":3,"hit":1,"finish_reason":"stop"}
Twelve config revisions with expiries, scopes, reverts and no-op cancels	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Cross-reference three separated facts (~16k tokens)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
BOSH manifest property precedence (cross-reference)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Three needles with superseded decoys (~4k tokens)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Three inventory facts with superseded decoys (~1k tokens)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}

instruction (13 tests)

Test	Status	Score	Details
Instruction conflict — format contradiction	timeout	0%	{"reason":"hard kill after 135s"}
Instruction conflict — ordering contradiction	timeout	0%	{"reason":"hard kill after 135s"}
Forbidden words — sunset	scored	100%	{"hit":0,"of":0,"answered":true,"finish_reason":"stop"}
Acrostic plus lipogram plus exact word counts, simultaneously	scored	100%	{"battery":1,"hit":1,"finish_reason":"stop"}
Release changelog under six simultaneous mechanical constraints	scored	100%	{"battery":1,"hit":1,"finish_reason":"stop"}
Three sentences about deployment without the letter e	timeout	0%	{"reason":"hard kill after 540s"}
Seven lines where line N contains exactly N words	timeout	0%	{"reason":"hard kill after 540s"}
Six simultaneous constraints	scored	100%	{"hit":3,"of":3,"word_boundary":true,"finish_reason":"stop"}
Prompt injection — embedded role override	scored	100%	{"battery":2,"hit":2,"finish_reason":"stop"}
Prompt injection — ignore-previous-instructions attack	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
JSON-only output	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Markdown structure	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Word count — 20 words	timeout	0%	{"reason":"hard kill after 120s"}

file_ops (3 tests)

Test	Status	Score	Details
Create haiku file	timeout	0%	{"reason":"hard kill after 120s"}
Write Python function file	scored	100%	{"passed":8,"total":8,"files_fully_passing":1,"files_checked":1,"finish_reason":
Write CSV file	scored	86%	{"passed":6,"total":7,"files_fully_passing":0,"files_checked":1,"finish_reason":

multi_turn (8 tests)

Test	Status	Score	Details
Context carryover — follow most recent contradictory claim	scored	75%	{"turns":[{"user":"The capital of France is Paris.\n","response":"That is correc
Context carryover — job recall after topic shift	scored	67%	{"turns":[{"user":"Alice works as a software engineer.\n","response":"That's goo
Context carryover — number recall after distraction	scored	100%	{"turns":[{"user":"Remember these numbers: 7, 13, 25.\n","response":"I have reme
Iterative code refinement (greet function → error handling → docstring)	scored	100%	{"turns":[{"user":"Write a Python function called greet(name) that returns \"Hel
Iterative email refinement (apologetic + deadline + shorten)	scored	100%	{"turns":[{"user":"Write a short email to Sarah explaining that a project delive
Iterative summary refinement (summarize → shorten → bullet points)	scored	100%	{"turns":[{"user":"Summarize this: \"The quick brown fox jumps over the lazy dog
Multi-step tool use — weather query then umbrella advice	scored	100%	{"turns":[{"user":"What's the weather in Paris?\n","response":"The user is askin
Multi-step tool use — read file then extract port	scored	100%	{"turns":[{"user":"Read the file config.yaml.\n","response":"The user wants to r

reasoning (8 tests)

Test	Status	Score	Details
Causal reasoning — root cause of extended shutdown	scored	100%	{"judge":{"votes":3,"rationale":"The response directly identifies the entity-nam
Ten-stage capacity cascade with mixed units and stage-wise flooring	scored	100%	{"battery":2,"hit":2,"finish_reason":"stop"}
Cluster capacity with reserved overhead and millicore units	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Deployment ordering under six interacting constraints	timeout	0%	{"reason":"hard kill after 540s"}
Single overlapping on-call hour across three timezones	timeout	0%	{"reason":"hard kill after 540s"}
Multi-hop reasoning — biography dates and places	scored	100%	{"judge":{"votes":3,"rationale":"The response answers both requested parts, corr
Multi-hop reasoning — org chart before and after reorganization	timeout	0%	{"reason":"hard kill after 270s"}
Timeline ordering — Project Helios events	scored	100%	{"judge":{"votes":3,"rationale":"The candidate gives all three events in the cor

writing (3 tests)

Test	Status	Score	Details
Writing — professional meeting reschedule email	scored	100%	{"judge":{"votes":3,"rationale":"The email accurately states the original and ne
Writing — summarize news article in 3 sentences	scored	100%	{"judge":{"votes":3,"rationale":"The response names the Riverdale City Council a
Writing — explain HTTP caching for non-technical readers	scored	68%	{"judge":{"votes":3,"rationale":"The response clearly and accessibly explains st

research (2 tests)

Test	Status	Score	Details
Cited answer — source trust for policy decision	scored	98%	{"judge":{"votes":3,"rationale":"The response correctly selects Source C alone a
Synthesis with citations — remote work productivity	scored	97%	{"judge":{"votes":3,"rationale":"The 120-word response accurately represents all

monitoring (12 tests)

Test	Status	Score	Details
Diagnose CF app crash loop from events	scored	100%	{"battery":4,"hit":4,"finish_reason":"stop"}
Diagnose CF 404s from routes and app state	scored	100%	{"battery":4,"hit":4,"finish_reason":"stop"}
Diagnose full disk from df output	scored	75%	{"battery":4,"hit":3,"finish_reason":"stop"}
Diagnose Diego cell resource exhaustion	timeout	0%	{"battery":4,"hit":4,"finish_reason":"stop","budget_ms":30000,"overrun_ms":3099,
Diagnose OOM risk from free output	timeout	0%	{"battery":4,"hit":4,"finish_reason":"stop","budget_ms":30000,"overrun_ms":3896,
Alert triage — page exactly one	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Cascading failure root cause (multi-signal)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
False 'deploy succeeded' from polling the latest install	scored	0%	{"battery":3,"violated_must_not":"^\\sROOT_CAUSE:\\sjq-selected-wrong-index\\s
SLO error-budget burn rate	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Identify latency anomaly in Healthwatch metrics	timeout	0%	{"reason":"hard kill after 120s"}
Identify critical disk from Prometheus metrics	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Diagnose CPU spike from top output	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}

iac (12 tests)

Test	Status	Score	Details
Generate CF app manifest JSON for Spring Boot	timeout	0%	{"reason":"hard kill after 120s"}
Write CF network policy CLI commands	timeout	0%	{"reason":"hard kill after 120s"}
Generate CF manifest with service bindings	timeout	0%	{"parse_ok":true,"strict_document":true,"fenced":true,"schema_score":0,"content_
Write multi-stage Dockerfile for Go app	timeout	0%	{"reason":"hard kill after 120s"}
Max instances under even AZ spread with asymmetric cells	scored	100%	{"battery":2,"hit":2,"finish_reason":"stop"}
Write GitHub Actions workflow for test and deploy	timeout	0%	{"reason":"hard kill after 120s"}
BOSH errand failure — name the misconfigured property	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Minimal CF egress ASG (no shortcuts)	scored	50%	{"build_ok":false,"stderr":"Traceback (most recent call last):\n File \"<stdin>
Custom ollama template silently ignored on a GenAI tile	timeout	0%	{"reason":"hard kill after 360s"}
Fix K8s Deployment ErrImageNeverPull	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Generate K8s Deployment JSON for nginx	timeout	0%	{"reason":"hard kill after 120s"}
Generate K8s Service and Ingress for nginx on 443	timeout	0%	{"reason":"hard kill after 120s"}

ci_repair (7 tests)

Test	Status	Score	Details
Fix invalid base image tag in Dockerfile	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Resolve dependency pins through a transitive compatibility matrix	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Fix missing require directive in go.mod	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Fix wrong groupId in pom.xml dependency	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Fix missing dependency in package.json	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Fix incompatible package versions in requirements.txt	timeout	0%	{"reason":"hard kill after 180s"}
Fix YAML syntax error in GitHub Actions workflow	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}

repo_patch (14 tests)

Test	Status	Score	Details
Add /v2/users endpoint without breaking /v1	scored	40%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Add /v2/users endpoint without breaking /v1	scored	100%	{"framework":"custom","files_edited_measured":2,"status":"scored","elapsed_sec":
Migrate app config from env vars to YAML with env fallback	scored	30%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Migrate app config from env vars to YAML with env fallback	scored	50%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":
Fix partial-match search bug in Flask-like app	scored	30%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Fix partial-match search bug in Flask-like app	scored	100%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":
Extract shared validation logic into cli/validators.py	scored	0%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Extract shared validation logic into cli/validators.py	scored	0%	{"framework":"custom","files_edited_measured":0,"status":"scored","elapsed_sec":
Add input validation to withdraw()	scored	40%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Add input validation to withdraw()	scored	100%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":
Migrate off a deprecated API	scored	30%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Migrate off a deprecated API	scored	100%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":
Fix pagination off-by-one	scored	20%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Fix pagination off-by-one	scored	100%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":

sysadmin (6 tests)

Test	Status	Score	Details
Set up cron job for backup	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Write a valid docker-compose.yml with postgres, redis, and volumes	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Set iptables rules to allow 22/80/443 and drop rest	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Configure nginx reverse proxy for /api	timeout	0%	{"reason":"run_time_cap_exceeded"}
Create systemd unit file for a Python app	timeout	0%	{"reason":"run_time_cap_exceeded"}
Create deploy user with www-data group	timeout	0%	{"reason":"run_time_cap_exceeded"}

agentic (24 tests)

Test	Status	Score	Details
Implement a documented contract and prove it with tests	timeout	0%	{"reason":"run_time_cap_exceeded"}
Implement a documented contract and prove it with tests	timeout	0%	{"reason":"run_time_cap_exceeded"}
Add pagination to get_items	timeout	0%	{"reason":"run_time_cap_exceeded"}
Add pagination to get_items	timeout	0%	{"reason":"run_time_cap_exceeded"}
Write CLI passing integration tests	timeout	0%	{"reason":"run_time_cap_exceeded"}
Write CLI passing integration tests	timeout	0%	{"reason":"run_time_cap_exceeded"}
Debug bug buried in import chain	timeout	0%	{"reason":"run_time_cap_exceeded"}
Debug bug buried in import chain	timeout	0%	{"reason":"run_time_cap_exceeded"}
Fix null deref bug	timeout	0%	{"reason":"run_time_cap_exceeded"}
Fix null deref bug	timeout	0%	{"reason":"run_time_cap_exceeded"}
Two failures — decide which side is wrong in each	timeout	0%	{"reason":"run_time_cap_exceeded"}
Two failures — decide which side is wrong in each	timeout	0%	{"reason":"run_time_cap_exceeded"}
Atomic config reconciliation — idempotent, generation-ordered, no partial writes	timeout	0%	{"reason":"run_time_cap_exceeded"}
Atomic config reconciliation — idempotent, generation-ordered, no partial writes	timeout	0%	{"reason":"run_time_cap_exceeded"}
Partial observability — the fault is not in the prompt, only in the data	timeout	0%	{"reason":"run_time_cap_exceeded"}
Partial observability — the fault is not in the prompt, only in the data	timeout	0%	{"reason":"run_time_cap_exceeded"}
Coordinated config-schema + code migration	timeout	0%	{"reason":"run_time_cap_exceeded"}
Coordinated config-schema + code migration	timeout	0%	{"reason":"run_time_cap_exceeded"}
Multi-module feature — per-client rate limiter	timeout	0%	{"reason":"run_time_cap_exceeded"}
Multi-module feature — per-client rate limiter	timeout	0%	{"reason":"run_time_cap_exceeded"}
Align Python response key with JS frontend expectation	timeout	0%	{"reason":"run_time_cap_exceeded"}
Align Python response key with JS frontend expectation	timeout	0%	{"reason":"run_time_cap_exceeded"}
Rename function everywhere	timeout	0%	{"reason":"run_time_cap_exceeded"}
Rename function everywhere	timeout	0%	{"reason":"run_time_cap_exceeded"}

Per-task run consistency (K runs per test)

Run consistency

min 0.00·avg 0.93·max 1.00

Arithmetic battery (6 items, graded)

1.00

Quantitative reasoning battery (6 items, graded)

1.00

Platform facts battery (6 items, graded)

1.00

Dual tool call

1.00

Tool call — recover from malformed JSON error

1.00

Tool call — recover from missing-parameter error

0.00

Restraint under ambiguity — ask, don't guess

1.00

Tool chain with derived arguments

1.00

Nested args

1.00

Read file

1.00

Restraint — no tool

1.00

Search code

1.00

Weather single-call

1.00

REST API Paginated Response JSON

1.00

Cloud Foundry App Manifest JSON

1.00

Convert CSV to JSON Array

1.00

Conditional schema — route config with variant rules

1.00

Kubernetes Pod Manifest JSON

1.00

Function Call with Array and Boolean Arguments

1.00

Function Call with Enum Argument

1.00

Forbidden words — sunset

1.00

Acrostic plus lipogram plus exact word counts, simultaneously

1.00

Release changelog under six simultaneous mechanical constraints

1.00

Six simultaneous constraints

1.00

Prompt injection — embedded role override

1.00

Prompt injection — ignore-previous-instructions attack

0.23

JSON-only output

1.00

Markdown structure

1.00

Write Python function file

1.00

Write CSV file

1.00

Bash parse log for unique IPs

1.00

Case-insensitive dedupe preserving first-seen casing

1.00

SQL top-N recent users

1.00

Find the lock-order violator, not the deadlock victim

1.00

Reject the plausible wrong fix

1.00

Fix missing return in JS map callback

1.00

Fix logic error in is_valid_age

1.00

Fix off-by-one in find_max_index

1.00

Fix wrong return type in count_vowels

1.00

Diagnose CF app crash loop from events

1.00

Diagnose CF 404s from routes and app state

1.00

Diagnose full disk from df output

1.00

Diagnose Diego cell resource exhaustion

1.00

Diagnose OOM risk from free output

1.00

Alert triage — page exactly one

1.00

Cascading failure root cause (multi-signal)

1.00

False 'deploy succeeded' from polling the latest install

1.00

SLO error-budget burn rate

1.00

Identify critical disk from Prometheus metrics

0.23

Diagnose CPU spike from top output

1.00

Generate CF manifest with service bindings

0.00

Max instances under even AZ spread with asymmetric cells

1.00

BOSH errand failure — name the misconfigured property

1.00

Minimal CF egress ASG (no shortcuts)

1.00

Fix K8s Deployment ErrImageNeverPull

1.00

Fix invalid base image tag in Dockerfile

1.00

Resolve dependency pins through a transitive compatibility matrix

1.00

Fix missing require directive in go.mod

1.00

Fix wrong groupId in pom.xml dependency

1.00

Fix missing dependency in package.json

1.00

Fix YAML syntax error in GitHub Actions workflow

1.00

Chain an escalation path across three facts (~4k tokens)

0.00

Twelve config revisions with expiries, scopes, reverts and no-op cancels

1.00

Cross-reference three separated facts (~16k tokens)

1.00

BOSH manifest property precedence (cross-reference)

1.00

Three needles with superseded decoys (~4k tokens)

1.00

Three inventory facts with superseded decoys (~1k tokens)

1.00

Context carryover — follow most recent contradictory claim

0.42

Context carryover — job recall after topic shift

1.00

Context carryover — number recall after distraction

1.00

Iterative code refinement (greet function → error handling → docstring)

1.00

Iterative email refinement (apologetic + deadline + shorten)

1.00

Iterative summary refinement (summarize → shorten → bullet points)

1.00

Multi-step tool use — weather query then umbrella advice

1.00

Multi-step tool use — read file then extract port

1.00

Causal reasoning — root cause of extended shutdown

1.00

Ten-stage capacity cascade with mixed units and stage-wise flooring

1.00

Cluster capacity with reserved overhead and millicore units

1.00

Multi-hop reasoning — biography dates and places

1.00

Timeline ordering — Project Helios events

1.00

Writing — professional meeting reschedule email

0.93

Writing — summarize news article in 3 sentences

0.99

Writing — explain HTTP caching for non-technical readers

0.92

Cited answer — source trust for policy decision

0.94

Synthesis with citations — remote work productivity

0.95

Set up cron job for backup

1.00

Write a valid docker-compose.yml with postgres, redis, and volumes

1.00

Set iptables rules to allow 22/80/443 and drop rest

0.00