Study Finds Moderate LoRA Ranks Efficient for Diffusion Fine-Tuning

Summary

The paper presents a controlled study of how LoRA rank affects diffusion-model fine-tuning quality and resource use. Using CIFAR-10, a DDPM U-Net, fixed optimization settings, and ranks 2, 4, 8, 16, and 32, the authors measure FID, trainable parameters, runtime, and GPU memory with a reproducible local-folder PyTorch-FID protocol. Rank 4 produces the best reported DDPM FID at 124.1380, while rank 8 is close at 124.2136. Increasing the rank beyond these moderate settings yields limited quality gains despite increasing adaptation cost. The authors also test extended-budget DDPM runs for 20 epochs at ranks 4, 8, and 16, and a Tiny DiT backbone for 10 epochs at the same ranks to check whether the observed trend extends beyond the initial experiment. They conclude that small-to-moderate LoRA ranks are practical defaults when training budgets are fixed.