TrimSFT Trims Token-Level Supervision for Mathematical Reasoning

Summary

The paper examines whether supervised fine-tuning for mathematical reasoning should apply the same cross-entropy pressure to every target token. It argues that uniform SFT can over-train tokens the model already predicts confidently while placing excessive pressure on tokens that the current model weakly supports. The authors propose Trimmed Logit-Gap SFT, or TrimSFT, which weights each token according to the logit gap between the gold token and its strongest competitor. The method reduces supervision at both extremes: large-gap tokens that are already mastered and small or negative-gap tokens that are weakly supported, while emphasizing an intermediate range. Its implementation uses a Gaussian weight centered on a margin and controlled by a bandwidth, and requires neither a reference model nor an additional forward pass. Experiments cover six base models from the Llama, Qwen, and DeepMath families and five mathematical reasoning benchmarks. TrimSFT consistently outperforms standard SFT, records the best average result on five of the six models, and improves MATH500 by as much as 26.9 points over SFT. Further analysis finds that the bandwidth matters more than the exact margin location, while variants that trim only one side produce worse trade-offs. Token-level logit-gap distributions also suggest that the two-sided method reshapes confidence more evenly than uniform or monotonic reweighting approaches.