Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

llama.cpp-omni

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. OpenBMB's Lookahead Expert Scheduling runs a 35B MoE at 20.5 tok/s on iPhone

    A post on r/LocalLLaMA relays an OpenBMB engineering write-up describing Lookahead Expert Scheduling (LES), a technique used with OpenBMB's MiniCPM-V 4.7 to run a 35B-A3B mixture-of-experts model locally without changing its routing or dropping experts. According to the relayed write-up, LES predicts which experts the next layer may need, preloads their weights from flash storage while current computations run, and executes ready experts first. Combined with optimized weight layouts, mixed-precision quantization and DSpark speculative decoding, the post cites 20.51 tok/s on an iPhone 18 Pro Max — 6.07x a 3.38 tok/s baseline — plus expert weights reported 26.7% smaller with a 0.16-point drop across five benchmarks and around 3 GB peak memory in a low-memory configuration. The implementation, including expert scheduling and DSpark integration, is said to be open source in llama.cpp-omni.

    r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my llama.cpp-omni experienceAsk about llama.cpp-omni

Nobody has shared their experience with llama.cpp-omni yet.