创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

OpenBMB 专家预调度:iPhone 本地跑 35B MoE

AI 总结

r/LocalLLaMA 上一篇帖子转述 OpenBMB 的工程文章,介绍名为 Lookahead Expert Scheduling(LES)的技术;OpenBMB 的 MiniCPM-V 4.7 借此在本地运行 35B-A3B 混合专家模型,且不改变实际路由、不丢弃专家。按转述内容,LES 会预测下一层可能需要的专家,在当前计算进行时从闪存预加载其权重,并优先执行已就绪的专家。配合优化的权重布局、混合精度量化与 DSpark 投机解码,帖子称在 iPhone 18 Pro Max 上达到 20.51 tok/s,为 3.38 tok/s 基线的 6.07 倍;专家权重缩小 26.7%,在五项基准上仅下降 0.16 分,低内存配置下峰值内存约 3 GB。帖子还称包括专家调度与 DSpark 集成在内的实现在 llama.cpp-omni 中开源。

深度见解

发帖者认为,LES 在不改变模型实际执行哪些专家的前提下掩盖了闪存读取瓶颈,并认为这一思路或可适用于其他 MoE 模型。

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called Lookahead Expert Scheduling (LES) to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts. The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first. Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves: The implementation, including expert scheduling and DSpark integration, is open source in llama.cpp-omni. The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too. submitted by /u/GasSmooth7439 [link] [comments]

#OpenBMB#MiniCPM-V 4.7#llama.cpp-omni#DSpark#iPhone 18 Pro Max

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。