0% Success, Zero Exceptions — a Score Too Clean to Trust

Weeks of 'the model just isn't very good' numbers from an AI-agent benchmark turned out to be measuring nothing at all — the grading script itself was silently crashing on every single run, including the ones that actually succeeded.

July 31, 2026

This article is also available in Traditional Chinese: 中文版 — same content, just a language difference.

Running an agent-coding benchmark against a local model on an ARM64 AI workstation, the pass rate sat at a suspiciously round 0% for far longer than felt right. Zero exceptions, zero partial credit, nothing in the summary output that looked like an error — just a clean, uniform failure that read exactly like "the model isn't good enough yet." It wasn't that. The benchmark's own scoring machinery was the thing that was broken, and it had been broken since before the first number was ever trusted.

The first false lead: assuming it's a hardware problem

The setup: a standard agent-coding benchmark suite, built around x86_64 Docker images, running against a local model on an ARM64 AI workstation — meaning every benchmark container runs under CPU emulation, not natively. The first instinct when trials started coming back slow and failing was to suspect the hardware itself was underpowered for the job, and to start pricing out a second, beefier machine. A quick look at actual load average (a small fraction of available cores in use) and GPU utilization (idle) ruled that out immediately — this wasn't a capacity problem, so throwing more hardware at it wouldn't have fixed anything.

Unlock this article to keep reading, or subscribe for unlimited access to everything. See Pricing for details.