Evaluating AI Model Capabilities After Benchmark Saturation

Summary

As AI capabilities advance, benchmarks can lose their ability to distinguish models once scores approach perfection, weakening their value for safety cases and frontier policy. This study asks whether the full transcripts produced while solving benchmark items retain useful capability information. The researchers retrospectively simulate saturation using historical benchmark and model releases, then test whether protocols using only correctly solved transcripts can predict performance on held-out items from benchmarks that have not saturated. They compare pairwise judging, single-transcript absolute scoring, and deterministic transcript features or embeddings. Saturated transcripts do retain capability-relevant information, and pairwise judging performs best among the tested methods, reducing prediction error by 5.3 times compared with a baseline based only on saturated benchmark scores. The useful signal is spread across transcript sections but is concentrated in channels such as reasoning and tool calls. However, the methods do not always generalize across distribution shifts: for example, they underestimated the capabilities of the first reasoning models introduced in early 2025. The findings suggest that saturated benchmarks can remain useful if robust methods are developed to extract information from their transcripts, rather than relying on accuracy scores alone.