Extremely Sparse Supervision Can Incentivize LLM Reasoning

Summary

Large language models are often post-trained with objectives covering very large numbers of generated tokens, based on the assumption that effective learning is token-intensive. This study revisits that assumption in the on-policy distillation setting, where a teacher can supervise every token generated by a student. Using the Qwen3 family, the authors find that supervising only one or two tokens in a reasoning trajectory, about 0.05% of all tokens, can effectively improve reasoning. In most tested cases, this sparse objective matches or surpasses training on every token, despite excluding nearly all generated tokens from optimization. The result appears consistently across nine teacher-student configurations with different model scales on mathematical reasoning tasks. Additional validation covers coding reasoning, Llama models, and PPO-based reinforcement learning with verifiable rewards. The authors suggest that a few critical reasoning steps may provide a more efficient learning signal than word-by-word correction, while presenting the finding as a direction for designing and understanding more efficient post-training algorithms.