AlphaDiana Introduces a Measurement System for Agents in Verifiable Reasoning Tasks

Summary

AlphaDiana is a measurement system for studying AI agents that combine reasoning models with harnesses for model calls, tool execution, skill instructions, and memory. It provides modular interfaces for datasets, agents, environments, and scorers, while preserving each harness's native control loop and separately controlling file reuse and memory retention. The system supports both whole-agent analysis and matched micro-interventions on tools, skills, and memory. The authors evaluated three open-source models with three open-source harnesses on five answer-based benchmarks, and tested two models on two interactive benchmarks. Their results show that harnesses do not consistently improve accuracy: gains on questions a model cannot solve alone may be offset by losses on questions it answers correctly without tools. Skill guidance and retained history likewise did not reliably improve accuracy. The study also finds that behavior and output-token use must be interpreted alongside correctness, because low token entropy can accompany wrong answers and fewer tokens per execution can produce more tokens per correct answer when accuracy declines. AlphaDiana connects correctness and execution status with traces and resource records under shared task and resource policies, enabling comparisons and failure diagnosis.