TheoPhysEval Tests AI Agents on Autonomous Theoretical Physics

Summary

TheoPhysEval introduces a suite of 50 unpublished, research-level theoretical physics problems with verifiable answers. The authors say a typical problem could be solved by a world expert in about a day or less, while requiring substantially longer for experts outside the relevant specialty. They evaluate off-the-shelf AI agents autonomously, without human hints, while allowing realistic tools such as literature search and scripts for numerical and symbolic work. The best-performing system, GPT-6-Astra running in Codex, reaches 98% final-answer accuracy at an average cost of under $3 per problem. Removing web search and access to background papers causes only small average accuracy changes, which the authors interpret as evidence of substantial memorization of the literature. The results indicate that current agents can accurately carry out concrete calculations across theoretical physics research. However, the paper also addresses a gap between strong performance on defined problems and the absence of agent-led breakthroughs, discussing several possible explanations without claiming that the benchmark resolves that question.