Janas-LLM Runs 80B MoE Models on a 32GB Laptop, Outpacing llama.cpp in Tests

Summary

Janas is an experimental family of C tools whose Janas-LLM component runs large mixture-of-experts language models on ordinary computers with a CPU, GPU, or both. Its main technique keeps model weights on disk, streams the experts needed for each token from an NVMe SSD, and retains frequently used experts in memory. On a Debian 13 laptop with an Intel Core Ultra 9 185H, integrated Arc GPU, 32 GB of RAM, and an NVMe SSD, the project reports 37 tokens per second for Qwen3-Next-80B-A3B with drafts, or 24 without drafts, while llama.cpp produced 2.8 tokens per second without drafts; prompt reading was 62 versus 16 tokens per second. Across 13 Q4_K_M model comparisons, Janas-LLM was faster than llama.cpp on the same machine, although one 3-bit Qwen3-30B-A3B variant read prompts more slowly while generating three times faster. The benchmark covers models from Qwen and Gemma ranging from 0.6B to 80B parameters, and reports results as means of repeated rounds for most tests. Draft-based acceleration depends on the model and text: reported draft acceptance ranged from about one third to 89 percent, and some models do not provide a drafting block. Janas also reports similar generation speeds on Windows 11, with different prompt-reading results depending on the model and driver. The project supports Linux x86-64 and Windows 10 or later, provides pre-converted model files, and includes chat, HTTP serving, MCP, code-completion, library, conversion, and benchmarking tools. It is explicitly early-stage and unverified: its cache can consume free memory, heavy disk access can make the desktop unresponsive, and GPU use has once caused a driver reset. Results are hardware-specific, and the author says other CPUs, GPUs, disks, and memory configurations remain largely untested.