The Math Behind AI Models: Wilkinson’s Backward Error

Summary

Modern AI systems depend on enormous numbers of floating-point operations, but floating-point arithmetic was once viewed with deep suspicion because repeated rounding could make computed answers unreliable. This explainer examines how James Wilkinson, who received the Turing Award in 1970, changed that understanding through backward-error analysis. Rather than comparing a computed result with a known correct answer, which is often unavailable, backward error asks how much the input must be perturbed for the computed result to become exactly correct. For a well-designed algorithm, that perturbation can remain close to the scale of rounding error, sometimes around 10^-16. Wilkinson’s approach became a standard way to assess numerical algorithms, while also clarifying that small backward error does not guarantee a small final error. Sensitive or “finicky” inputs can amplify errors regardless of the algorithm or machine used; this sensitivity is measured by the condition number. The central relationship is that forward error is approximately the condition number multiplied by backward error. The article presents this framework as a basis for trusting the floating-point calculations underlying modern AI systems.