A Technical Tour of 2026’s Open-Weight LLM Architectures

Summary

Sebastian Raschka surveys the main open-weight LLM releases from January through early March 2026, focusing on how their architectures differ and what those choices imply for inference efficiency. The review covers Arcee AI’s Trinity Large, Moonshot’s Kimi K2.5, StepFun’s Step 3.5 Flash, Qwen3-Coder-Next, GLM-5, MiniMax M2.5, Nanbeige 4.1 3B, Qwen3.5, Ant Group’s Ling and Ring 2.5, Cohere’s Tiny Aya, and the later Sarvam 30B and 105B models. The models range from small systems designed for local use to MoE models with hundreds of billions or one trillion total parameters. Kimi K2.5 adds vision through joint pretraining on roughly 15 trillion mixed visual and text tokens, while Qwen3.5 also brings multimodal support to its main model line. Several releases use hybrid attention to reduce long-context memory or compute: Qwen3-Coder-Next combines Gated DeltaNet with gated attention, Qwen3.5 continues that design, and Ling 2.5 uses Lightning Attention with DeepSeek-style MLA. Trinity uses alternating local and global attention, QK-Norm, gated attention, and depth-scaled normalization. Step 3.5 Flash combines a 196B MoE with 11B active parameters and multi-token prediction, and reports 100 tokens per second at 128K context compared with 33 for DeepSeek V3.2 in the cited comparison. GLM-5 scales to 744B parameters and adopts MLA and DeepSeek Sparse Attention, while MiniMax M2.5 emphasizes a smaller 230B design and coding performance. Tiny Aya is a 3.35B multilingual model with a parallel transformer block, but its license permits only non-commercial use. Sarvam’s 105B model uses MLA, while its 30B version uses conventional GQA; the team reports higher throughput and strong performance on Indian languages. Across the survey, the author argues that architecture matters for efficiency and capability, but dataset quality and training recipes also strongly influence model performance. The review presents recurring design trade-offs rather than identifying one universally best architecture.