PerfReasoning Benchmarks LLM Hardware-Performance Reasoning

Summary

PerfReasoning is a benchmark for testing large language models on hardware-performance reasoning and on generating analytical performance-model code. Given workload, architecture, and mapping specifications, models must compare mappings and predict off-chip traffic and buffer requirements. The strongest closed-source models score above 90% on reasoning-based questions, while the best open-weight model reaches 82.4%. Constructing the models is much harder: GPT-5.6 Sol exceeds an 80% pass rate, but every other tested configuration averages below 15%, with substantial variation across runs. Task-specific reinforcement learning improves mapping-reasoning accuracy for a 4B model by 15.7 percentage points. By contrast, feedback-free multi-round self-revision prompting is not reliably effective. The benchmark therefore exposes a gap between producing plausible architectural reasoning and reliably turning that reasoning into executable performance models. The authors plan to release PerfReasoning publicly to support reproducible evaluation and future progress tracking.