Five-Bugs Smoke Test Finds Better Tests Matter More Than Bigger Coding Models

Summary

The GitHub repository five-bugs presents a short, reproducible smoke test for AI coding agents. Five single-file Python bugs are paired with a checker visible to the agent and a second checker written afterward; the experiment asks whether models can solve what the tests fail to specify. Claude Code and Codex CLI passed all five visible checkers, but both failed the hidden checker for task four in exactly the same way. The seeded defect was a greedy HTML-stripping regex, and the hidden case contained literal comparison characters in text; every attempted replacement still transformed “a < b and c > d” into “a d.” Across 26 attempts, ranging from a 4-bit quantized Qwen2.5-Coder 7B to Claude Haiku 4.5, Sonnet 5 and Opus 5, four regex patterns appeared and none passed the hidden test. Ten independent runs on the weak specification all failed that case, while exposing the hidden checker as the visible specification made all five tasks pass, including task four. The repository therefore supports a narrower conclusion: on these localized, textbook bugs, the test suite constrained the answer more strongly than model scale, although the author warns that five tasks, one laptop and one attempt per cell cannot generalize to large refactors or unfamiliar debugging. The archived results also compare tools, model tiers, output compression and quantization. Claude Code’s median time was 21.3 seconds versus 29.6 seconds for Codex CLI, with wider task-level variation for Codex. Haiku, Sonnet and Opus all scored 5/5 on visible tests and 4/5 on hidden tests, while cost rose from $0.28 to $1.53. A quantized Qwen2.5-Coder 7B scored 24/40 visibly and 16/40 on hidden tests under both tested quantizations, despite producing different answers and taking different times. The repository includes raw CSV files, patches, methods and verification scripts; `scripts/verify.py` replays archived patches without running agents or spending quota.