A Four-Stage Mechanistic Account of LLM Math Reasoning and Distractor Failures

Summary

Large language models can solve grade-school math word problems accurately, but adding a single irrelevant clause can cause their performance to collapse. This paper develops a mechanistic explanation by identifying a sequential four-stage internal pipeline: Schema Abstraction, Operation Planning, Operand Binding, and Computation. Each stage produces a distinct intermediate representation that appears in an identifiable band of model layers. Applying the same analytical scaffold to problems containing distractors, the authors localize the resulting corruption to Operation Planning rather than to the later stages. They further identify a set of attention heads involved in that stage and validate its causal role through bidirectional interventions. The work therefore connects observable robustness failures with a specific internal computation, offering a mechanistic interpretation of both successful math word-problem reasoning and its breakdown under distraction.