Moritz Hardt Argues AI Benchmarking Should Become a Science

Summary

Computer scientist Moritz Hardt argues that the AI benchmarking ecosystem is fundamentally broken, even though benchmarks helped shape the field. He says benchmark scores are often unstable because test sets are public and can enter model-development loops, making absolute accuracy difficult to interpret. Historically, however, relative rankings were often stable when models were trained on the same data, allowing benchmarks to identify genuine progress despite the lack of a strong theory explaining why. Hardt believes that assumption no longer holds for large language models because developers’ training data and post-training procedures are opaque, and models can be optimized for particular tests, or “benchmaxxed.” His proposed remedy is to let every model undergo a common period of benchmark-relevant preparation before evaluation, an approach he compares with altitude training for athletes. This would not make absolute scores scientifically meaningful, but could make rankings fairer by reducing differences in how much each model was prepared for a test. Hardt also cautions that benchmarking measures progress more reliably than it explains how models work. He argues that benchmarks can still contribute to scientific knowledge, citing the CASP protein-folding benchmark as an example of progress preceding a complete theory.