How New LLM Architectures Reduce the Cost of Long Context

Summary

Sebastian Raschka surveys architecture changes in recent open-weight large language models, focusing on mechanisms that make long-context inference cheaper rather than on training data, post-training, benchmarks, or product comparisons. Gemma 4 E2B and E4B reuse key-value tensors across layers: later layers retain their own queries but share KV projections with earlier layers of the same attention type. Because roughly half of the KV tensors are shared, the article estimates savings of about 2.7 GB for Gemma 4 E2B and 6 GB for E4B at bfloat16 precision and a 128K context, while noting that sharing reduces model capacity. The same Gemma variants use per-layer embeddings, adding token-specific capacity through small layer-specific vectors without scaling the full transformer stack; their effective and embedding-inclusive parameter counts are 2.3B versus 5.1B for E2B and 4.5B versus 8B for E4B. Laguna XS.2 uses layer-wise attention budgeting across 40 layers: 30 sliding-window layers with a 512-token window and 10 global layers, with different query-head counts while keeping KV heads fixed. ZAYA1-8B introduces Compressed Convolutional Attention, which performs attention directly on compressed Q, K, and V representations and adds convolutional mixing to compressed Q and K; the cited CCA experiments report an advantage over MLA under comparable compression, though this is not presented as universally superior. DeepSeek V4 combines manifold-constrained hyper-connections, which widen the residual pathway into interacting streams with stability constraints, with Compressed Sparse Attention and Heavily Compressed Attention. HCA compresses every 128 tokens into one KV entry, while CSA uses milder compression, sparse selection, and a local uncompressed window. Relative to DeepSeek V3.2 at a 1M-token context, the article reports 27% of single-token inference FLOPs and 10% of KV-cache size for V4-Pro, and 10% and 7% respectively for V4-Flash. The author cautions that the comparison reflects the full model recipes, lacks an ablation study, and does not establish CSA/HCA as generally better than MLA. Overall, the surveyed designs preserve the decoder-only transformer while adding increasingly specialized mechanisms to reduce memory, attention cost, or compute, at the price of greater implementation complexity.