Understanding and Implementing Qwen3 From Scratch

Summary

Sebastian Raschka presents a hands-on guide to Qwen3, an open-weight large language model family initially released in May and updated in July. The article implements the model’s dense and Mixture-of-Experts architectures from scratch in pure PyTorch so readers can inspect the main building blocks and adapt them for experiments or projects. Raschka highlights Qwen3’s Apache License 2.0, which he describes as developer- and commercially friendly, as one reason for its popularity. He also points to the family’s reported performance: the 235B-Instruct variant was ranked eighth on the LMArena leaderboard at the time of writing, tied with Claude Opus 4, while only DeepSeek 3.1 and Kimi K2 ranked higher among the open-weight models discussed. The family spans model sizes from 0.6B dense models to 480B-parameter Mixture-of-Experts models, covering different compute budgets and use cases. The article is structured as a long code-focused tutorial intended to explain how Qwen3 works under the hood rather than only describe it conceptually. It also notes a 1T-parameter Qwen3 “max” variant released on the provider’s platform, while stating that this variant was closed-source at the time described.