Beyond Standard LLMs: Alternative Architectures for Efficiency and Reasoning

Summary

Sebastian Raschka surveys alternatives to the decoder-style autoregressive Transformers that still dominate open-weight LLMs. The article first examines linear-attention hybrids such as MiniMax-M1, Qwen3-Next, DeepSeek V3.2, and Kimi Linear, which replace many quadratic attention layers with recurrent or subquadratic mechanisms while retaining some full-attention layers. Qwen3-Next uses a 3:1 mix of Gated DeltaNet and gated attention and supports a native 262k-token context; Kimi Linear uses Kimi Delta Attention and multi-head latent attention, and is reported to reduce KV-cache use by 75% and deliver up to 6x decoding throughput versus regular full attention. The trade-off is reduced global-context capacity or reasoning accuracy in some settings, illustrated by MiniMax M2 returning to regular attention after the team found linear attention difficult for reasoning and multi-turn production workloads. Text diffusion models take a more radical route: they mask and iteratively unmask tokens in parallel rather than generate them one at a time. This may reduce generation steps for long outputs, but ParallelBench reported quality degradation under parallel decoding, while tool use, streaming, and the speed-quality trade-off remain unresolved. Code World Models add execution traces to training so that a model learns to predict program states, not only plausible code. The 32-billion-parameter CWM reportedly matched gpt-oss-20b and, with best@k test-time scaling, slightly exceeded gpt-oss-120b on the cited coding evaluation while using four times fewer parameters. Finally, HRM and the 7-million-parameter Tiny Recursive Model repeatedly refine latent states and answers on grid-based tasks such as ARC and Sudoku. TRM improved over HRM in the reported comparisons, including a Sudoku ablation from 79.5% to 87.4% after reducing layers, but these models remain specialized puzzle solvers rather than general-purpose LLMs. Raschka concludes that standard autoregressive Transformers remain the practical default, while hybrid attention is especially promising for long contexts and the other approaches are research directions with clear constraints.