How Modern LLM Architectures Are Evolving: From DeepSeek V3 to GLM-5

Summary

Sebastian Raschka surveys the architectural design of major open-weight large language models released from DeepSeek V3 through GLM-5 and Gemma 4, focusing on structure rather than training data or benchmark rankings. The article explains how Multi-Head Latent Attention (MLA) compresses key-value representations for smaller KV caches, while Mixture-of-Experts (MoE) increases total model capacity but activates only a subset of experts per token. DeepSeek V3, for example, has 671 billion total parameters but activates 37 billion during inference; its MLA and shared-expert design are contrasted with the GQA-based MoE design of Llama 4 and the dense and sparse variants of Qwen3. Other models pursue efficiency through different mechanisms: Gemma 3 uses a 5:1 mix of sliding-window and global attention with a 1,024-token local window, while Gemma 4 retains this structure and adds key reuse and partial RoPE. OLMo 2 and Olmo 3 use post-normalization and QK-Norm, and SmolLM3 applies NoPE in selected layers, although the generalization of NoPE results from smaller experiments remains uncertain. Qwen3-Next, Kimi Linear, and Nemotron 3 combine attention with Gated DeltaNet or Mamba-style state-space components to reduce long-context memory and computation, trading some precise retrieval capability for efficiency. The article also discusses multi-token prediction, latent experts, sparse attention, and model width-versus-depth choices. Across the comparison, MoE, hybrid attention, KV-cache reduction, and increasingly specialized normalization or gating recur as major design directions, but the author cautions that architecture alone cannot explain model quality because training methods, data, and hyperparameters vary widely. The technical comparison therefore identifies design patterns and tradeoffs rather than claiming a single best architecture.