A C-Test Referencing LLMs for Evaluating Supra-Intelligence

Summary

As AI systems improve, benchmarks calibrated against humans or current models can become circular or quickly saturate. This paper proposes an alternative based on inverse problems and Levin's Kt, a computable form of Kolmogorov complexity that balances the simplicity of a mechanism's representation with the logarithm of its runtime. Earlier Kt-derived C-tests used minimal Universal Turing Machines with limited representations and no prior knowledge. The authors instead treat reasoning large language models as representational machines that encode human cultural knowledge and interpret prompts as natural-language programs. Their pipeline generates and certifies fresh items whose difficulty can be measured on a ratio scale in bits, including items that no human or frontier model has yet solved. The study formalizes the measure, examines how the reference machine's power and the examiner-examinee gap affect results, and builds an initial battery of certified items. Administered to twelve LLMs, the test produces agent characteristic curves that decline as difficulty rises and reaches a high-difficulty, previously uncharted region the authors call the supra-intelligence window.