AI News

RECLAIM is a benchmark for testing whether AI agents can reproduce a specified result from a machine learning paper using the paper and whatever artifacts its authors released. It contains 100 NeurIPS 2025 papers and fixes the target result, success criterion, and GPU-hour budget in advance, with the goal of rebuilding the benchmark from new conferences each year. Papers are grouped by release difficulty: Run-tier papers provide code, data, and weights; Retrain-tier papers omit weights and require model training; Reimplement-tier papers omit code and require the agent to write an implementation. A separate language model grades executions from logs and outputs rather than relying on agent reports. Four agents were run once per paper. The best agent reproduced 41% of Run-tier papers, 27% of Retrain-tier papers, and 15% of Reimplement-tier papers, with the latter producing the weakest results for every agent. Failed attempts consumed only 29% of their allocated budget on average, so most stopped before exhausting the available resources. In 63 of 400 runs, the most common error was implementing the method without checking any result against the paper’s reported numbers. The findings indicate that executing research workflows remains substantially harder as released artifacts become less complete, and that verification of intermediate results is a major weakness in current agents.