创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

BigMoeOnEdge公布本地MoE推理测试

AI 总结

BigMoeOnEdge开发者称,该本地推理工具可加载llama.cpp支持的任意MoE GGUF模型,无需重新打包或使用特殊格式。它支持NVIDIA、AMD、Intel、Mac、手机和CPU,并覆盖Windows、Linux、macOS、Android与iOS,还可用--auto自动配置机器。开发者测试显示其在8至16GB设备上表现突出,但在双RTX 5060 Ti运行Qwen3.8-Flash-Next时,Strata快1.3倍。

深度见解

BigMoeOnEdge为本地运行MoE模型提供了广泛兼容的选择,但其性能和输出质量目前主要依据开发者自测。

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data

submitted by /u/dai_app [link] [comments]

#BigMoeOnEdge#llama.cpp#Strata#colibri#Basalt#Qwen3.6-35B#Qwen3.8-Flash-Next#NVIDIA GeForce RTX 5060 Ti

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。