Transformers streams SSD-hosted experts for 180B MoE inference
Tarek Ziadé reported a Transformers implementation that streams quantized MoE experts from SSD into a bounded device cache, enabling the 180B-parameter Qwen3.8-Flash-Next model to run on a MacBook M5 with 32 GB of unified memory. The author measured 13.9 tokens per second after optimization and said this remained about 6% behind Strata, with per-layer synchronization identified as the remaining gap. The implementation is available through an open pull request, but its API was still under discussion with maintainers.
The work provides a concrete local-inference approach for running a large MoE model within consumer laptop memory, while its self-reported performance and unfinished API leave room for independent validation.
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
Published November 18, 2026
#Hugging Face Transformers#Qwen3.8-Flash-Next#Strata#PyTorch#MacBook M5#llama.cpp#vLLM#Accelerate