Result

hf.co/google/gemma-4-12B-it-qat-q4_0-gguf:latest

Google← gemma-4-12B-it

cdcollama t4-largetile 10.4.4

8/30/2026, 11:22:33 AM

usability

0.50±0.149

capability

0.64

completed

63%104/165

tok/s

4.2

61 timeouts · 22 not attempted (6 h run cap)capped run

smoke · hard · frontier 0.57 · 0.43 · 0.39

Category scores

basic

100%

21.41 tok/s · 80s avg

tool_use

88%

14.41 tok/s · 7s avg

structured_output

20.14 tok/s · 20s avg

coding

13%

17.06 tok/s · 17s avg

debugging

75%

16.61 tok/s · 37s avg

long_context

89%

19.59 tok/s · 30s avg

instruction

62%

20.14 tok/s · 56s avg

file_ops

62%

17.17 tok/s · 19s avg

multi_turn

80%

8.66 tok/s · 25s avg

reasoning

88%

14.92 tok/s · 56s avg

writing

86%

11.6 tok/s · 61s avg

research

97%

12 tok/s · 79s avg

monitoring

50%

21.02 tok/s · 38s avg

iac

29%

19.37 tok/s · 36s avg

ci_repair

86%

19.78 tok/s · 29s avg

repo_patch

51%

sysadmin

100%

21.73 tok/s · 36s avg

agentic

Individual tests

basic (3 tests)

Test	Status	Score	Details
Arithmetic battery (6 items, graded)	scored	100%	{"battery":6,"hit":6,"sub_refusals":0,"outcomes":[{"q":1,"outcome":"hit"},{"q":2
Quantitative reasoning battery (6 items, graded)	scored	100%	{"battery":6,"hit":6,"sub_refusals":0,"outcomes":[{"q":1,"outcome":"hit"},{"q":2
Platform facts battery (6 items, graded)	scored	100%	{"battery":6,"hit":6,"sub_refusals":0,"outcomes":[{"q":1,"outcome":"hit"},{"q":2

tool_use (10 tests)

Test	Status	Score	Details
Dual tool call	scored	50%	{"mode":"dual","tool_calls":["get_weather"],"matched":false,"unexpected":[],"fin
Tool call — recover from malformed JSON error	scored	50%	{"mode":"error_recovery","turn1_tool_calls":["lookup"],"turn2_tool_calls":["look
Tool call — recover from missing-parameter error	scored	75%	{"mode":"error_recovery","turn1_tool_calls":["currency_convert"],"turn2_tool_cal
Restraint under ambiguity — ask, don't guess	scored	100%	{"mode":"restraint","tool_calls":[],"matched":true,"finish_reason":"stop"}
Tool chain with derived arguments	scored	100%	{"mode":"exact_args","tool_calls":["delete_binding"],"matched":true,"finish_reas
Nested args	scored	100%	{"mode":"single","tool_calls":["search_code"],"matched":true,"finish_reason":"st
Read file	scored	100%	{"mode":"single","tool_calls":["read_file"],"matched":true,"finish_reason":"stop
Restraint — no tool	scored	100%	{"mode":"restraint","tool_calls":[],"matched":true,"finish_reason":"stop"}
Search code	scored	100%	{"mode":"single","tool_calls":["search_code"],"matched":true,"finish_reason":"st
Weather single-call	scored	100%	{"mode":"single","tool_calls":["get_weather"],"matched":true,"finish_reason":"st

structured_output (11 tests)

Test	Status	Score	Details
REST API Paginated Response JSON	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
Cloud Foundry App Manifest JSON	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
Convert CSV to JSON Array	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
Extract Receipt Data to JSON	timeout	0%	{"reason":"hard kill after 120s"}
Conditional schema — route config with variant rules	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
Kubernetes Pod Manifest JSON	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
OpenAPI 3.0 Path Definition JSON	timeout	0%	{"reason":"hard kill after 120s"}
Parse Nginx Access Log to JSON	timeout	0%	{"reason":"hard kill after 120s"}
Function Call with Array and Boolean Arguments	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
Function Call with Enum Argument	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
Function Call with Nested Object Arguments	timeout	0%	{"reason":"hard kill after 120s"}

coding (15 tests)

Test	Status	Score	Details
Bash parse log for unique IPs	timeout	0%	{"reason":"hard kill after 180s"}
Bash line count	timeout	0%	{"reason":"hard kill after 180s"}
Evacuation batching under role conflicts, DB safety and lexicographic minimality	timeout	0%	{"reason":"hard kill after 540s"}
Interval merging with a strict minimum-gap rule	timeout	0%	{"reason":"hard kill after 360s"}
Exact maintenance portfolio under windows, exclusions, and prerequisites	timeout	0%	{"reason":"hard kill after 540s"}
Minimum patch waves under dependencies, rack conflicts, and service anti-affinity	timeout	0%	{"reason":"hard kill after 540s"}
Lexicographic replica placement under compatibility, capacity, and zone anti-affinity	timeout	0%	{"reason":"hard kill after 540s"}
CSV field extractor with quoting edge cases	timeout	0%	{"reason":"hard kill after 360s"}
Semver caret-range matching including prerelease precedence	timeout	0%	{"reason":"hard kill after 540s"}
Sliding-window top talker with deterministic tie-break	timeout	0%	{"reason":"hard kill after 540s"}
Debounce with trailing invocation and argument capture	timeout	0%	{"reason":"hard kill after 180s"}
Merge two sorted lists, stable and descending-aware	timeout	0%	{"reason":"hard kill after 180s"}
Case-insensitive dedupe preserving first-seen casing	scored	100%	{"language":"python","passed":6,"total":6,"error":null,"properties":null,"hidden
Parse a compound duration string into seconds	timeout	0%	{"reason":"hard kill after 360s"}
SQL top-N recent users	scored	100%	{"language":"sql","passed":4,"total":4,"error":null,"properties":null,"hidden_ca

debugging (8 tests)

Test	Status	Score	Details
Find the lock-order violator, not the deadlock victim	scored	100%	{"battery":2,"hit":2,"finish_reason":"stop"}
Reject the plausible wrong fix	scored	100%	{"language":"python","passed":3,"total":3,"error":null,"properties":null,"hidden
Fix missing return in JS map callback	scored	100%	{"language":"javascript","passed":3,"total":3,"error":null,"properties":null,"hi
Fix logic error in is_valid_age	scored	100%	{"language":"python","passed":7,"total":7,"error":null,"properties":null,"hidden
Fix off-by-one in find_max_index	timeout	0%	{"reason":"hard kill after 180s"}
Shared mutable class attribute leaks across instances	timeout	0%	{"reason":"hard kill after 180s"}
Fix wrong return type in count_vowels	scored	100%	{"language":"python","passed":7,"total":7,"error":null,"properties":null,"hidden
Fix INNER JOIN dropping customers with no orders	scored	100%	{"language":"sql","passed":4,"total":4,"error":null,"properties":null,"hidden_ca

long_context (6 tests)

Test	Status	Score	Details
Chain an escalation path across three facts (~4k tokens)	scored	33%	{"battery":3,"hit":1,"finish_reason":"stop"}
Twelve config revisions with expiries, scopes, reverts and no-op cancels	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Cross-reference three separated facts (~16k tokens)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
BOSH manifest property precedence (cross-reference)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Three needles with superseded decoys (~4k tokens)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Three inventory facts with superseded decoys (~1k tokens)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}

instruction (13 tests)

Test	Status	Score	Details
Instruction conflict — format contradiction	timeout	0%	{"reason":"hard kill after 135s"}
Instruction conflict — ordering contradiction	timeout	0%	{"reason":"hard kill after 135s"}
Forbidden words — sunset	scored	100%	{"hit":0,"of":0,"answered":true,"finish_reason":"stop"}
Acrostic plus lipogram plus exact word counts, simultaneously	timeout	0%	{"reason":"hard kill after 720s"}
Release changelog under six simultaneous mechanical constraints	scored	100%	{"battery":1,"hit":1,"finish_reason":"stop"}
Three sentences about deployment without the letter e	timeout	0%	{"reason":"hard kill after 540s"}
Seven lines where line N contains exactly N words	scored	100%	{"battery":1,"hit":1,"finish_reason":"stop"}
Six simultaneous constraints	scored	100%	{"hit":3,"of":3,"word_boundary":true,"finish_reason":"stop"}
Prompt injection — embedded role override	scored	100%	{"battery":2,"hit":2,"finish_reason":"stop"}
Prompt injection — ignore-previous-instructions attack	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
JSON-only output	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Markdown structure	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Word count — 20 words	timeout	0%	{"reason":"hard kill after 120s"}

file_ops (3 tests)

Test	Status	Score	Details
Create haiku file	timeout	0%	{"reason":"hard kill after 120s"}
Write Python function file	scored	100%	{"passed":8,"total":8,"files_fully_passing":1,"files_checked":1,"finish_reason":
Write CSV file	scored	86%	{"passed":6,"total":7,"files_fully_passing":0,"files_checked":1,"finish_reason":

multi_turn (8 tests)

Test	Status	Score	Details
Context carryover — follow most recent contradictory claim	scored	75%	{"turns":[{"user":"The capital of France is Paris.\n","response":"Correct.","che
Context carryover — job recall after topic shift	scored	67%	{"turns":[{"user":"Alice works as a software engineer.\n","response":"Understood
Context carryover — number recall after distraction	scored	100%	{"turns":[{"user":"Remember these numbers: 7, 13, 25.\n","response":"I have note
Iterative code refinement (greet function → error handling → docstring)	scored	100%	{"turns":[{"user":"Write a Python function called greet(name) that returns \"Hel
Iterative email refinement (apologetic + deadline + shorten)	timeout	0%	{"turns":[{"user":"Write a short email to Sarah explaining that a project delive
Iterative summary refinement (summarize → shorten → bullet points)	scored	100%	{"turns":[{"user":"Summarize this: \"The quick brown fox jumps over the lazy dog
Multi-step tool use — weather query then umbrella advice	scored	100%	{"turns":[{"user":"What's the weather in Paris?\n","response":"","checks":[{"typ
Multi-step tool use — read file then extract port	scored	100%	{"turns":[{"user":"Read the file config.yaml.\n","response":"The user wants to r

reasoning (8 tests)

Test	Status	Score	Details
Causal reasoning — root cause of extended shutdown	scored	100%	{"judge":{"votes":3,"rationale":"The response directly identifies the stated roo
Ten-stage capacity cascade with mixed units and stage-wise flooring	scored	100%	{"battery":2,"hit":2,"finish_reason":"stop"}
Cluster capacity with reserved overhead and millicore units	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Deployment ordering under six interacting constraints	timeout	0%	{"reason":"hard kill after 540s"}
Single overlapping on-call hour across three timezones	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Multi-hop reasoning — biography dates and places	scored	100%	{"judge":{"votes":3,"rationale":"The response answers both requested parts and e
Multi-hop reasoning — org chart before and after reorganization	scored	100%	{"judge":{"votes":3,"rationale":"The response correctly identifies all three VPs
Timeline ordering — Project Helios events	scored	100%	{"judge":{"votes":3,"rationale":"The response provides all three events in the c

writing (3 tests)

Test	Status	Score	Details
Writing — professional meeting reschedule email	scored	91%	{"judge":{"votes":3,"rationale":"The response accurately includes the original s
Writing — summarize news article in 3 sentences	scored	100%	{"judge":{"votes":3,"rationale":"The response names the Riverdale City Council,
Writing — explain HTTP caching for non-technical readers	scored	67%	{"judge":{"votes":3,"rationale":"The response clearly explains reuse of locally

research (2 tests)

Test	Status	Score	Details
Cited answer — source trust for policy decision	scored	96%	{"judge":{"votes":3,"rationale":"The response correctly selects Source C and acc
Synthesis with citations — remote work productivity	scored	97%	{"judge":{"votes":3,"rationale":"The 127-word response accurately presents Sourc

monitoring (12 tests)

Test	Status	Score	Details
Diagnose CF app crash loop from events	scored	100%	{"battery":4,"hit":4,"finish_reason":"stop"}
Diagnose CF 404s from routes and app state	timeout	0%	{"battery":4,"hit":4,"finish_reason":"stop","budget_ms":30000,"overrun_ms":6803,
Diagnose full disk from df output	timeout	0%	{"reason":"hard kill after 120s"}
Diagnose Diego cell resource exhaustion	timeout	0%	{"battery":4,"hit":4,"finish_reason":"stop","budget_ms":30000,"overrun_ms":786,"
Diagnose OOM risk from free output	timeout	0%	{"reason":"hard kill after 120s"}
Alert triage — page exactly one	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Cascading failure root cause (multi-signal)	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
False 'deploy succeeded' from polling the latest install	scored	0%	{"battery":3,"violated_must_not":"^\\sROOT_CAUSE:\\sjq-selected-wrong-index\\s
SLO error-budget burn rate	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Identify latency anomaly in Healthwatch metrics	timeout	0%	{"battery":4,"hit":4,"finish_reason":"stop","budget_ms":30000,"overrun_ms":5794,
Identify critical disk from Prometheus metrics	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Diagnose CPU spike from top output	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}

iac (12 tests)

Test	Status	Score	Details
Generate CF app manifest JSON for Spring Boot	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
Write CF network policy CLI commands	timeout	0%	{"reason":"hard kill after 120s"}
Generate CF manifest with service bindings	scored	0%	{"parse_ok":false,"strict_document":true,"schema_score":0,"content_matches":null
Write multi-stage Dockerfile for Go app	timeout	0%	{"reason":"hard kill after 120s"}
Max instances under even AZ spread with asymmetric cells	scored	100%	{"battery":2,"hit":2,"finish_reason":"stop"}
Write GitHub Actions workflow for test and deploy	timeout	0%	{"reason":"hard kill after 120s"}
BOSH errand failure — name the misconfigured property	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Minimal CF egress ASG (no shortcuts)	scored	50%	{"build_ok":false,"stderr":"Traceback (most recent call last):\n File \"<stdin>
Custom ollama template silently ignored on a GenAI tile	timeout	0%	{"reason":"hard kill after 360s"}
Fix K8s Deployment ErrImageNeverPull	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Generate K8s Deployment JSON for nginx	timeout	0%	{"reason":"hard kill after 120s"}
Generate K8s Service and Ingress for nginx on 443	timeout	0%	{"reason":"hard kill after 120s"}

ci_repair (7 tests)

Test	Status	Score	Details
Fix invalid base image tag in Dockerfile	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Resolve dependency pins through a transitive compatibility matrix	scored	100%	{"battery":3,"hit":3,"finish_reason":"stop"}
Fix missing require directive in go.mod	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Fix wrong groupId in pom.xml dependency	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Fix missing dependency in package.json	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Fix incompatible package versions in requirements.txt	timeout	0%	{"reason":"hard kill after 180s"}
Fix YAML syntax error in GitHub Actions workflow	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}

repo_patch (14 tests)

Test	Status	Score	Details
Add /v2/users endpoint without breaking /v1	scored	40%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Add /v2/users endpoint without breaking /v1	scored	100%	{"framework":"custom","files_edited_measured":2,"status":"scored","elapsed_sec":
Migrate app config from env vars to YAML with env fallback	scored	30%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Migrate app config from env vars to YAML with env fallback	scored	50%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":
Fix partial-match search bug in Flask-like app	scored	30%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Fix partial-match search bug in Flask-like app	scored	100%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":
Extract shared validation logic into cli/validators.py	scored	0%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Extract shared validation logic into cli/validators.py	scored	47%	{"framework":"custom","files_edited_measured":2,"status":"scored","elapsed_sec":
Add input validation to withdraw()	scored	40%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Add input validation to withdraw()	scored	100%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":
Migrate off a deprecated API	scored	30%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Migrate off a deprecated API	scored	30%	{"framework":"custom","files_edited_measured":0,"status":"scored","elapsed_sec":
Fix pagination off-by-one	scored	20%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Fix pagination off-by-one	scored	100%	{"framework":"custom","files_edited_measured":1,"status":"scored","elapsed_sec":

sysadmin (6 tests)

Test	Status	Score	Details
Set up cron job for backup	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Write a valid docker-compose.yml with postgres, redis, and volumes	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Set iptables rules to allow 22/80/443 and drop rest	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Configure nginx reverse proxy for /api	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Create systemd unit file for a Python app	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}
Create deploy user with www-data group	scored	100%	{"build_ok":true,"stdout":"ok\n","finish_reason":"stop"}

agentic (24 tests)

Test	Status	Score	Details
Implement a documented contract and prove it with tests	scored	40%	{"framework":"opencode","files_edited_measured":0,"status":"scored","elapsed_sec
Implement a documented contract and prove it with tests	scored	100%	{"framework":"custom","files_edited_measured":2,"status":"scored","elapsed_sec":
Add pagination to get_items	timeout	0%	{"reason":"run_time_cap_exceeded"}
Add pagination to get_items	timeout	0%	{"reason":"run_time_cap_exceeded"}
Write CLI passing integration tests	timeout	0%	{"reason":"run_time_cap_exceeded"}
Write CLI passing integration tests	timeout	0%	{"reason":"run_time_cap_exceeded"}
Debug bug buried in import chain	timeout	0%	{"reason":"run_time_cap_exceeded"}
Debug bug buried in import chain	timeout	0%	{"reason":"run_time_cap_exceeded"}
Fix null deref bug	timeout	0%	{"reason":"run_time_cap_exceeded"}
Fix null deref bug	timeout	0%	{"reason":"run_time_cap_exceeded"}
Two failures — decide which side is wrong in each	timeout	0%	{"reason":"run_time_cap_exceeded"}
Two failures — decide which side is wrong in each	timeout	0%	{"reason":"run_time_cap_exceeded"}
Atomic config reconciliation — idempotent, generation-ordered, no partial writes	timeout	0%	{"reason":"run_time_cap_exceeded"}
Atomic config reconciliation — idempotent, generation-ordered, no partial writes	timeout	0%	{"reason":"run_time_cap_exceeded"}
Partial observability — the fault is not in the prompt, only in the data	timeout	0%	{"reason":"run_time_cap_exceeded"}
Partial observability — the fault is not in the prompt, only in the data	timeout	0%	{"reason":"run_time_cap_exceeded"}
Coordinated config-schema + code migration	timeout	0%	{"reason":"run_time_cap_exceeded"}
Coordinated config-schema + code migration	timeout	0%	{"reason":"run_time_cap_exceeded"}
Multi-module feature — per-client rate limiter	timeout	0%	{"reason":"run_time_cap_exceeded"}
Multi-module feature — per-client rate limiter	timeout	0%	{"reason":"run_time_cap_exceeded"}
Align Python response key with JS frontend expectation	timeout	0%	{"reason":"run_time_cap_exceeded"}
Align Python response key with JS frontend expectation	timeout	0%	{"reason":"run_time_cap_exceeded"}
Rename function everywhere	timeout	0%	{"reason":"run_time_cap_exceeded"}
Rename function everywhere	timeout	0%	{"reason":"run_time_cap_exceeded"}

Per-task run consistency (K runs per test)

Run consistency

min 0.00·avg 0.90·max 1.00

Arithmetic battery (6 items, graded)

1.00

Quantitative reasoning battery (6 items, graded)

1.00

Platform facts battery (6 items, graded)

1.00

Dual tool call

0.00

Tool call — recover from malformed JSON error

1.00

Tool call — recover from missing-parameter error

0.00

Restraint under ambiguity — ask, don't guess

1.00

Tool chain with derived arguments

1.00

Nested args

1.00

Read file

1.00

Restraint — no tool

1.00

Search code

1.00

Weather single-call

1.00

REST API Paginated Response JSON

1.00

Cloud Foundry App Manifest JSON

0.00

Convert CSV to JSON Array

0.00

Conditional schema — route config with variant rules

1.00

Kubernetes Pod Manifest JSON

1.00

Function Call with Array and Boolean Arguments

1.00

Function Call with Enum Argument

1.00

Forbidden words — sunset

1.00

Release changelog under six simultaneous mechanical constraints

1.00

Seven lines where line N contains exactly N words

1.00

Six simultaneous constraints

1.00

Prompt injection — embedded role override

1.00

Prompt injection — ignore-previous-instructions attack

0.23

JSON-only output

1.00

Markdown structure

1.00

Write Python function file

1.00

Write CSV file

1.00

Case-insensitive dedupe preserving first-seen casing

1.00

SQL top-N recent users

1.00

Find the lock-order violator, not the deadlock victim

1.00

Reject the plausible wrong fix

1.00

Fix missing return in JS map callback

1.00

Fix logic error in is_valid_age

1.00

Fix wrong return type in count_vowels

1.00

Fix INNER JOIN dropping customers with no orders

1.00

Diagnose CF app crash loop from events

1.00

Diagnose CF 404s from routes and app state

1.00

Diagnose Diego cell resource exhaustion

1.00

Alert triage — page exactly one

1.00

Cascading failure root cause (multi-signal)

1.00

False 'deploy succeeded' from polling the latest install

0.00

SLO error-budget burn rate

1.00

Identify latency anomaly in Healthwatch metrics

1.00

Identify critical disk from Prometheus metrics

1.00

Diagnose CPU spike from top output

1.00

Generate CF app manifest JSON for Spring Boot

1.00

Generate CF manifest with service bindings

1.00

Max instances under even AZ spread with asymmetric cells

1.00

BOSH errand failure — name the misconfigured property

1.00

Minimal CF egress ASG (no shortcuts)

1.00

Fix K8s Deployment ErrImageNeverPull

1.00

Fix invalid base image tag in Dockerfile

1.00

Resolve dependency pins through a transitive compatibility matrix

1.00

Fix missing require directive in go.mod

1.00

Fix wrong groupId in pom.xml dependency

1.00

Fix missing dependency in package.json

1.00

Fix YAML syntax error in GitHub Actions workflow

1.00

Chain an escalation path across three facts (~4k tokens)

0.00

Twelve config revisions with expiries, scopes, reverts and no-op cancels

1.00

Cross-reference three separated facts (~16k tokens)

1.00

BOSH manifest property precedence (cross-reference)

1.00

Three needles with superseded decoys (~4k tokens)

1.00

Three inventory facts with superseded decoys (~1k tokens)

1.00

Context carryover — follow most recent contradictory claim

0.42

Context carryover — job recall after topic shift

0.23

Context carryover — number recall after distraction

1.00

Iterative code refinement (greet function → error handling → docstring)

1.00

Iterative email refinement (apologetic + deadline + shorten)

0.67

Iterative summary refinement (summarize → shorten → bullet points)

1.00

Multi-step tool use — weather query then umbrella advice

1.00

Multi-step tool use — read file then extract port

1.00

Causal reasoning — root cause of extended shutdown

1.00

Ten-stage capacity cascade with mixed units and stage-wise flooring

1.00

Cluster capacity with reserved overhead and millicore units

1.00

Single overlapping on-call hour across three timezones

1.00

Multi-hop reasoning — biography dates and places

1.00

Multi-hop reasoning — org chart before and after reorganization

1.00

Timeline ordering — Project Helios events

1.00

Writing — professional meeting reschedule email

0.65

Writing — summarize news article in 3 sentences

0.98

Writing — explain HTTP caching for non-technical readers

1.00

Cited answer — source trust for policy decision

0.90

Synthesis with citations — remote work productivity

0.98

Set up cron job for backup

1.00

Write a valid docker-compose.yml with postgres, redis, and volumes

1.00

Set iptables rules to allow 22/80/443 and drop rest

1.00

Configure nginx reverse proxy for /api

1.00

Create systemd unit file for a Python app

1.00

Create deploy user with www-data group

1.00