Open-Weight Models Match Closed Models in CTI-REALM at a Fraction of the Cost

Summary

This study examines the trade-off between open-weight and closed language-model agents for security work using CTI-REALM, a benchmark that asks an agent to turn a threat-intelligence report into a working detection by querying a live telemetry store. The researchers evaluate 26 models, including 19 open-weight and seven closed models, with a shared harness, scorer, and panel of language-model judges. Their main result is that an open-weight model can match the strongest closed models for less than one-tenth of the cost when it receives enough turns. They identify a confound in the benchmark's published message limit: tasks that exhaust the limit receive zero at every checkpoint, including checkpoints already passed, so the limit partly measures whether a model finishes in time. At that limit, the leading open-weight and closed models are statistically indistinguishable in the reported results; after the limit is relaxed, truncated models recover and open-weight models obtain the highest rewards. Among leading models, similar rewards can correspond to costs differing by more than an order of magnitude, making price a poor capability proxy. The study also introduces stage-level metrics that execute submitted detection queries in the telemetry store and compare outputs with an answer key, rather than relying mainly on what agents say they did. Because different models lead different stages, the authors suggest routing each workflow stage to its strongest model instead of selecting one model by overall reward. Finally, the share of output devoted to thinking does not track reward.