Measuring the Inference Energy Efficiency of Open-Weight LLMs

Summary

Benchmark scores show what a large language model can do, but not the energy required to deliver that capability. This study directly measures 86 open-weight model checkpoints across seven benchmarks and six hardware platforms, from a consumer GPU to an eight-GPU server. Models with similar capability can differ substantially in efficiency: most tested configurations use a median of 2.6 to 4.8 times more energy than another configuration scoring at least as well, with the gap reaching 27 times. Enabling reasoning or thinking raises energy use by about 3.7 times on average, although the capability benefit varies by model family and output budget. Generated-token count is an imperfect energy proxy and misorders 18% to 24% of model pairs. FP8 lowers energy by roughly a quarter with little score change, while hardware mainly rescales energy without changing the relative model ranking. The authors argue that benchmark reporting should include measured energy under a specified serving regime and output budget.