llama.cpp-omni
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- OpenBMB's Lookahead Expert Scheduling runs a 35B MoE at 20.5 tok/s on iPhone
A post on r/LocalLLaMA relays an OpenBMB engineering write-up describing Lookahead Expert Scheduling (LES), a technique used with OpenBMB's MiniCPM-V 4.7 to run a 35B-A3B mixture-of-experts model locally without changing its routing or dropping experts. According to the relayed write-up, LES predicts which experts the next layer may need, preloads their weights from flash storage while current computations run, and executes ready experts first. Combined with optimized weight layouts, mixed-precision quantization and DSpark speculative decoding, the post cites 20.51 tok/s on an iPhone 18 Pro Max — 6.07x a 3.38 tok/s baseline — plus expert weights reported 26.7% smaller with a 0.16-point drop across five benchmarks and around 3 GB peak memory in a low-memory configuration. The implementation, including expert scheduling and DSpark integration, is said to be open source in llama.cpp-omni.
r/LocalLLaMA · 🔥 0
Experience and discussion from the community
Share my llama.cpp-omni experienceAsk about llama.cpp-omni
Nobody has shared their experience with llama.cpp-omni yet.