Researchers from the University of California, Berkeley, and MIT have released FreeToken, an open-source inference engine designed to run frontier mixture-of-experts models on consumer hardware. The project includes contributors such as Databricks co-founder Matei Zaharia, Ion Stoica, Song Han, and Kurt Keutzer. Its central problem is that sparse MoE models activate only part of their parameters per token, but decoding still requires routing through very large pools of expert weights. On consumer systems, limited PCIe bandwidth and host-memory latency can make static CPU-to-GPU expert offloading stall the GPU whenever a cache miss requires a weight transfer. FreeToken's q* strategy instead assigns token computation dynamically between CPU cores and GPU tensor cores according to current interconnect throughput. Its fast weight format and full-layer double buffering overlap PCIe transfers with active-layer computation, while an elastic memory manager adjusts the balance between KV-cache entries and resident expert slots without reloading the model. For agent workloads, semantic-anchor checkpoints preserve intermediate attention states and loop activations so that changes to tool parameters or inserted execution results can reuse valid subsequences instead of invalidating the entire prompt cache. The article reports that FreeToken decodes three to four times faster and prefills six to 30 times faster than Ollama and llama.cpp on the same MoE model, while differing from the data-center assumptions of vLLM and SGLang and the static offloading rules of KTransformers. Reported tests include about 39 tokens per second for Qwen3.6-35B on an 8GB RTX 4060 laptop, plus runs of DeepSeek-V4-Flash at 284B parameters and GLM-5.2 at 753B parameters on systems with an RTX 5090 or a single workstation GPU. The command-line tool and desktop client are available through FlashML.ai and GitHub for Linux and Windows systems using NVIDIA RTX 30, 40, and 50 series GPUs. Community discussions support interest in local hardware autonomy but question whether the theoretical q* allocation captures real CPU scheduling delays, memory contention, and changing expert residency under concurrent agent workloads.
