Sebastian Raschka’s Curated List of LLM Research Papers for January–May 2026

Summary

Sebastian Raschka presents a curated reference list of large language model research papers bookmarked from January through May 2026. He explicitly says it is not a complete record of papers published during the period: it reflects papers he considered useful or interesting for his own writing, teaching, coding, and research, and he examined some papers in detail while relying on titles, abstracts, and topic framing for others. The list is organized into categories including architecture and model design, efficient training and scaling, inference efficiency and KV caches, sparse attention and long context, reasoning and test-time compute, reinforcement learning, agent systems and tool use, coding agents, diffusion language models, and evaluation. Compared with his 2025 lists, he includes more work on agent harnesses, tool use, long context, diffusion language models, and serving infrastructure, while retaining a strong emphasis on reasoning models, reinforcement learning, and efficient inference. The architecture section highlights research that moves beyond simply scaling Transformers, including hybrid architectures, state-space layers, mixture-of-experts capacity allocation, activation behavior, and representation geometry. As a detailed example, the author discusses Nemotron 3 Super, a 120B-A12B model that alternates regular attention with Mamba-2 layers to improve efficiency on long contexts. He notes that the model may be too large for typical consumer hardware, while a 4B Nemotron 3 Nano variant is available. Nvidia had also released Nemotron 3 Ultra, a 550B-A55B scaled-up version using the same basic building blocks. The article connects this design direction with other hybrid open-weight models, including Qwen3.6, which uses Gated DeltaNet layers instead of Mamba-2 for its non-attention components. It also points to additional Nemotron 3 investigations involving multi-token prediction, NVFP4 versus BF16 pretraining, synthetic MMLU-style data, and post-training quantization, while leaving those analyses outside the overview’s scope.