Four Main Approaches to Evaluating Large Language Models

Summary

Sebastian Raschka presents a practical overview of four common approaches for evaluating large language models: multiple-choice benchmarks, verification-based evaluation, preference leaderboards, and LLM judges. Multiple-choice tests such as MMLU offer standardized, reproducible accuracy measurements across 57 subjects and about 16,000 questions, but mainly measure knowledge recall and selection from predefined options. Verifiers allow free-form answers and can use tools such as calculators or code interpreters to check results, making them valuable for mathematics and code; their use is limited to domains with reliable ground truth, and outcome-only checks may miss reasoning quality. Preference leaderboards such as LM Arena rank models through pairwise human votes, capturing perceived helpfulness, style, and safety, but they are affected by user demographics, prompts, voting bias, and the difference between preference and correctness. The article explains Elo ratings with a small implementation, while noting that LM Arena has moved to the Bradley–Terry model, which estimates rankings jointly and can express uncertainty through confidence intervals. LLM-as-a-judge systems ask a separate model to grade a response against an instruction, reference answer, and rubric; they scale more easily than human voting and support free-form tasks, but depend on judge capability, rubric design, prompt choices, and response style. Process reward models are described as step-level judges for intermediate reasoning signals, primarily in training, although they can be difficult to develop reliably. For hands-on examples, the article evaluates a from-scratch Qwen3 0.6B implementation on MMLU and shows how to run OpenAI’s 20-billion-parameter open-weight gpt-oss locally through Ollama as a judge, with a smaller Qwen3 option for machines with less memory. The conclusion is that no single method is sufficient: evaluation should combine complementary measures and include task-specific data that reflects the intended use case.