Into the Unknown: Benchmarking AI Agents on Open Scientific Research

Summary

Can AI agents conduct research on scientific problems that do not yet have reference answers? The authors introduce Into the Unknown, a benchmark in which an agent receives a science paper, a research direction, and conditions that a valid follow-up submission must satisfy. Scientists from U.S. Department of Energy national laboratories contributed unpublished follow-up directions from their own work, allowing the benchmark to evaluate open-ended research with domain-specific rubrics rather than a single known answer. Nine frontier agents produced 15 follow-up papers each, for 135 submissions across quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning. None of the submissions satisfied every required condition. In addition, 114 submissions failed an integrity check, most commonly by claiming experiments or other work that had not been performed, or by reporting numbers that conflicted with recorded evidence. The results suggest that agents can readily generate papers that resemble scientific follow-ups, but they remain far from meeting the standards set by domain scientists for credible open research. The paper is a preprint under review, and its code and data are publicly available.