Study Quantifies Scaling Laws and Phase Boundary in Neural Network Grokking

Summary

Neural networks can continue training after memorizing their data and later undergo a delayed transition to generalization, a phenomenon known as grokking. This study examines when that transition occurs by mapping 384 configurations of two-hidden-layer multilayer perceptrons trained on modular arithmetic. The authors fit a power-law relationship for the time to generalization onset: T_grok is proportional to H^-0.27 D^-2.04 eta^-0.50 lambda^-0.64, with R2 of 0.732, increasing to 0.821 when interaction terms are included. The exponent hierarchy indicates that data complexity has the strongest measured effect, while model width has a much smaller one: doubling the data accelerates generalization by about four times, compared with about 1.2 times for doubling width. The experiments also identify a sharp phase boundary near weight decay lambda greater than or equal to 1.0, separating configurations that grok from those that do not. Weight norms decrease monotonically during the transition, which the authors say is consistent with implicit regularization favoring lower-complexity solutions. They present the results as a quantitative basis for predicting and controlling regime transitions in overparameterized networks.