Harness or Model? Measuring the Harness Effect in Agentic Coding

Summary

This paper examines whether an agentic coding system’s harness, rather than its underlying language model, explains the common assumption that vendor-native toolchains solve more tasks. The authors run paired same-model contrasts on a private, contamination-controlled suite containing 256 repository and post-cutoff contest tasks. Eighty tasks were run with Claude Opus 4.8 under the Claude Agent SDK and DeepAgents, and with GPT-5.5 under the OpenAI Codex SDK and DeepAgents; Gemini 3.5 Flash and DeepSeek-V3.2 appear as side cells. Of 800 planned runs, 792 were graded by an isolated oracle. Neither comparison establishes an average performance advantage: the native harness was 1.25 percentage points behind for Opus 4.8, while the OpenAI Codex SDK was 1.25 points ahead for GPT-5.5, with confidence intervals spanning zero. The Opus result differs sharply by task stratum: its native harness trailed by 9.0 points on 61 repository tasks but led by 23.7 points on 19 contest tasks. Because that partition was selected after seeing the data, the authors say it requires designed replication. The study also separates correctness from completion: 22 of 81 runs cancelled at the wall-clock limit had already produced a passing patch. Based on observed usage and frozen list prices, the neutral harness cost 1.3 to 1.6 times more per solved Opus task and 1.2 times more per solved GPT-5.5 task. However, 58 Anthropic-account runs had no usage record, leaving the billed cost ordering unresolved; allocating that missing spend produces an Opus cost ratio ranging from 0.7 to 2.3. This revision corrects an earlier manuscript whose cost analysis used faulty telemetry semantics. The authors release the orchestrator, grading oracle, reanalysis code, and derived aggregates, while keeping the tasks private.