What Does Rank Buy in LoRA? A Spectral and Distributional Analysis

Summary

This paper examines whether the rank r in Low-Rank Adaptation (LoRA) functions as a capacity control that makes smaller-rank models simpler and better at generalization. Under hard per-factor norm budgets, an idealization of practical weight decay and norm control, the reachable updates are exactly the rank-at-most-r matrices inside a nuclear-norm ball. The paper shows that the complexity and displacement functionals it studies are maximized by rank-one updates, so the rank constraint does not bind. Consequently, the linear-readout model class is identical for every r >= 1, its Rademacher complexity has no dependence on r, and the maximum displacement of the source distribution has a sharp rank-independent upper bound. The authors then identify two settings in which rank does matter. Replacing separate factor budgets with a joint budget on their product yields a data-dependent, rank-sensitive complexity bound, although the benefit appears only for well-spread feature distributions and the worst case remains rank-free. Spectrally, rank determines the price of adaptation: canceling leading singular directions of the pretrained weight requires enough rank and budget. The paper derives upper and lower bounds on the minimum rank needed for a target source-to-target alignment, with matching bounds under two-sided spectral decay. It concludes that, in this setting, rank governs which updates are reachable and how costly cancellation is, rather than overall model capacity.