Artificial Analysis Launches FrontierCode 1.1 Coding-Agent Benchmark

Summary

Artificial Analysis has launched FrontierCode 1.1, a coding benchmark from Cognition that tests whether AI agents can make changes maintainers of real open-source repositories would merge. The benchmark goes beyond passing tests: it also checks for excessive changes, oversized diffs, out-of-scope file edits, and unhelpful tests. The evaluation uses the 100-task FrontierCode 1.1 Main set, with three attempts per task, private tasks, and the same open-source Stirrup agent harness for every model; each attempt is capped at 250 turns. Hidden tests, command checks, scope and code-volume limits, and qualitative LLM review determine whether checks are blocking or non-blocking. An attempt that fails a blocking check scores zero, while one that passes all blocking checks can score from 70% to 100%; GPT-6 Luna serves as the judge. Claude Opus 5.5 with fallback leads with a 52% mean score and a 58% pass rate, followed by GPT-6.1 Sol and Claude Fable 5.1 at 49%. Nine models from six labs score between 43% and 52%, while MiMo-V2.6-Pro leads the open-weights models at 45%. Nine tasks are passed by no model, and no task is passed by every model on every attempt. GPT-6.1 Sol offers the strongest reported score-to-cost tradeoff at 49% and $0.81 per task; costs range from $0.09 to $12.00. The benchmark’s cost frontier includes GPT-6 Luna, MiMo-V2.6-Pro, GPT-6.1 Sol, and Claude Opus 5.5, with the final three score points costing more than 12 times as much per task. GPT-6 models use roughly 27,000 to 33,000 output tokens per task, compared with about 157,000 for Claude Opus 5.5 and 236,000 for Claude Sonnet 5.5. Task times range from about three to 46 minutes. Artificial Analysis says FrontierCode will later be added to its Coding Agents leaderboard and is currently separate from its Intelligence Index.