Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

OpenBMB's Lookahead Expert Scheduling runs a 35B MoE at 20.5 tok/s on iPhone

AI summary

A post on r/LocalLLaMA relays an OpenBMB engineering write-up describing Lookahead Expert Scheduling (LES), a technique used with OpenBMB's MiniCPM-V 4.7 to run a 35B-A3B mixture-of-experts model locally without changing its routing or dropping experts. According to the relayed write-up, LES predicts which experts the next layer may need, preloads their weights from flash storage while current computations run, and executes ready experts first. Combined with optimized weight layouts, mixed-precision quantization and DSpark speculative decoding, the post cites 20.51 tok/s on an iPhone 18 Pro Max — 6.07x a 3.38 tok/s baseline — plus expert weights reported 26.7% smaller with a 0.16-point drop across five benchmarks and around 3 GB peak memory in a low-memory configuration. The implementation, including expert scheduling and DSpark integration, is said to be open source in llama.cpp-omni.

In-depth insight

The poster argues LES hides flash-storage memory bottlenecks without altering which experts the model actually executes, and suggests the approach could potentially work with other MoE models too.

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called Lookahead Expert Scheduling (LES) to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts. The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first. Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves: The implementation, including expert scheduling and DSpark integration, is open source in llama.cpp-omni. The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too. submitted by /u/GasSmooth7439 [link] [comments]

#OpenBMB#MiniCPM-V 4.7#llama.cpp-omni#DSpark#iPhone 18 Pro Max

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.