Harbor Adapters and Harbor-Index Build Infrastructure for Large-Scale Agent Evaluation

Summary

Evaluating language-model agents across many benchmarks is difficult because each benchmark may require a complex environment and a separate agent integration. The paper introduces Harbor Adapters, a unified infrastructure that ports more than 80 agentic benchmarks so they can evaluate arbitrary agents. The authors validate the adapters through code review and parity experiments. Using this infrastructure, they evaluate eight models spanning different capability tiers on 54 benchmarks, running each model with Terminus-2 and with one of three native harnesses. This setup supports a broader analysis of agent capabilities and failure modes than isolated benchmark runs. The authors also introduce Harbor-Index, a curated collection of 82 difficult and diverse tasks from 29 benchmarks. The collection was refined through difficulty filtering, AI and human audits, and an audit-and-fix process. It is intended to retain the challenge and breadth of large-scale evaluation while reducing the cost of running the suite. No evaluated model-harness configuration exceeded a 30% pass rate, and the strongest reported configuration, GPT-5.5 with Codex, achieved 28.0%. The authors release the adapters, results, detailed analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.