A Visual Guide to Attention Variants in Modern LLMs

Summary

Sebastian Raschka introduces an LLM architecture gallery with 45 visual model cards and uses the article to explain attention variants found in recent open-weight models. It begins with multi-head attention, where parallel query, key, and value projections let different heads learn different token relationships under a causal mask. Grouped-query attention shares key-value projections across several query heads, reducing KV-cache memory and traffic while preserving a relatively familiar implementation; it spans a spectrum from multi-query attention to full multi-head attention. Multi-head latent attention takes a different approach by caching compressed latent representations, and the article presents DeepSeek-V2 ablations suggesting that MLA can preserve or sometimes exceed MHA modeling quality more effectively than GQA at large scale, while noting that GQA may remain easier to tune for smaller models. Sliding-window attention limits each token to a recent local window, often alongside occasional global layers; Gemma 3 is cited as using a 5:1 local-to-global pattern with a 1,024-token window and only a small reported perplexity effect. DeepSeek Sparse Attention learns which prior tokens to revisit through an indexer and selector rather than using a fixed window, and can be combined with MLA. Gated attention adds output gating, normalization changes, and partial RoPE to retained full-attention layers. Hybrid architectures replace most full-attention layers with cheaper modules such as Gated DeltaNet, Kimi Delta Attention, Lightning Attention, or Mamba-2, while keeping heavier layers for content retrieval. The article describes Qwen3-Next and Qwen3.5, Kimi Linear, Ling 2.5, and Nemotron as examples, but concludes that no public controlled experiment establishes one universally best architecture; current choices depend on model scale, context length, implementation maturity, and the balance between quality and inference efficiency.