AI News

AI infrastructure has traditionally emphasized token generation, but the article argues that processing context before the first token is becoming an equally important bottleneck. It explains that prefill, which processes input context, is dominated by parallel matrix multiplication and is compute-bound, while decode, which generates output tokens, depends more heavily on memory bandwidth and data movement. Longer context windows, retrieval-augmented generation, multimodal inputs, and agentic workflows increase prefill demand, sometimes causing one request to trigger multiple model calls with accumulated context. The article cites frontier-lab projections of roughly 1,000 times more effective compute demand over five years, saying conventional digital accelerators could require an estimated $100 trillion in infrastructure investment and about 1,000 GW of additional electrical capacity to meet it. It identifies three related risks: fixed data-center power budgets, GPU capacity being diverted from token generation to context processing, and rising inference costs that could make long-context and agentic applications harder to price competitively. Lumai argues that prefill and decode should use specialized hardware rather than relying on the same accelerator for both workloads. Its Iris Nova system uses light instead of electricity for matrix multiplication and completes each vector-matrix multiplication in one optical cycle. In Lumai’s testing, the system ran billion-parameter language models in real time and delivered approximately 10 times more compute per watt than GPU-based equivalents on prefill workloads. The article presents this as a way to improve GPU utilization and energy efficiency without replacing all existing infrastructure, while acknowledging that future data centers may become increasingly heterogeneous. Iris Nova evaluations are available through Lumai, and the article also links to a Lumai white paper on prefill optimization.