GoldiMask Improves Fine-Tuning of Diffusion Language Models

Summary

Supervised fine-tuning of discrete diffusion language models requires deciding which response tokens remain visible as context and which are masked for prediction. The paper introduces GoldiMask, a method that approximately maximizes a submodular objective to select revealed tokens, balancing their contextual benefit against their value as learning targets. It then weights the remaining target tokens according to how much they benefit from the selected context and how much learning potential they retain. Across three model backbones and three training datasets, GoldiMask achieves the highest average accuracy in most evaluated settings, with gains reported for both reasoning and code generation. Ablation studies indicate that both context selection and target weighting contribute to the improvement. On GSM8K and MATH-500, GoldiMask also reduces decoding iterations when confidence-threshold parallel decoding is used, while retaining comparable accuracy at higher confidence thresholds.