Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

BigMoeOnEdge benchmarks MoE GGUF inference on local devices

AI summary

The BigMoeOnEdge developer says the local inference tool accepts any MoE GGUF model loadable by llama.cpp without repacking or a special format. It supports NVIDIA, AMD, Intel, Mac, phone and CPU hardware across Windows, Linux, macOS, Android and iOS, with an --auto option for configuring a machine. The developer's benchmarks report strong results on 8–16 GB devices, but Strata was 1.3 times faster on two RTX 5060 Ti cards running Qwen3.8-Flash-Next.

In-depth insight

BigMoeOnEdge offers a broadly compatible option for running MoE models locally, although its performance and output-quality claims currently rely on developer-supplied tests.

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data

submitted by /u/dai_app [link] [comments]

#BigMoeOnEdge#llama.cpp#Strata#colibri#Basalt#Qwen3.6-35B#Qwen3.8-Flash-Next#NVIDIA GeForce RTX 5060 Ti

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.