CogGym Enables Large-Scale Comparison of Human and AI Cognition

Summary

CogGym is a scalable framework for comparing human and model behavior on matched cognitive experiments. It uses a semi-automated, human-in-the-loop pipeline to convert diverse experiments into a task-agnostic Experiment Markup Language, enabling standardized and reproducible evaluation. The initial release organizes 258 experiments from 100 papers focused on human commonsense reasoning and evaluates 50 large language models against human responses. Larger and more recent models generally reproduce human judgments more closely. However, their gains on common-reasoning tasks are substantially slower than improvements reported on formal-reasoning benchmarks such as mathematics and coding. Model-human agreement also remains below human split-half reliability: human reliability is R²=0.93 for text, 0.95 for images, and 0.92 for video, while the best models reach 0.59, 0.58, and 0.43 respectively. The authors intend CogGym to become a living evaluation framework that adds new cognitive-science experiments as models and experiments evolve, helping track both human-like behavior and systematic divergences.