Benchmark Radar: A Living Database and Search Engine for AI Evaluations

Summary

Benchmark researchers and developers need to find relevant evaluations, locate associated datasets and code, and understand the settings behind reported scores. Benchmark Radar addresses this need as a living database and search engine covering LLM evaluation, agent and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. It combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable catalog, references to benchmark mentions in model cards and technical reports, and histories of reported scores. The system preserves source identities and citations so users can inspect candidate benchmarks and the evidence supporting evaluations. Its daily discovery pipeline draws on 37 sources, including 13 direct connectors and 24 first-party research and engineering feeds. The catalog contains 1,283 source records from four benchmark catalogs and 12,916 numeric observations covering 790 records. The paper describes the collection and retrieval process, audits the catalog, and analyzes benchmark saturation, adoption trends, and the limits of comparing scores. A worked example demonstrates a complete prior-art search, from querying the catalog to inspecting benchmark evidence when designing a new evaluation. The released web dashboard includes a benchmark leaderboard, a score-versus-measured-use Pareto view, saturation and trend views, daily feeds, downloadable evidence, and reproducible analysis. An accompanying command-line interface supports offline queries.