Basalt developer tests lossy MoE routing options
A Basalt developer reported an experiment with Qwen 3.8 Flash Next that tested two opt-in, lossy inference options: routing bias, which favors experts already on the GPU, and tail skip, which omits lower-ranked experts not resident on the GPU. Both reduce transfers from RAM but can change outputs. In a simulated small-card configuration, the shipped settings increased prose throughput by 56%, while ladder and fact scores fell by 5.2 and 2.3 points, respectively. The developer’s deliberately worst settings reduced fact accuracy from 38.1% to 2.7%. The test covered one model on one rig, used one seed for ladders and three for facts, and measured speed with single cold runs. Basalt ships both flags disabled by default and warns when they are enabled.
The report shows that speed-oriented MoE inference options can alter factual recall and chained reasoning, especially when a model does not fit fully in VRAM. It therefore supports checking whether such options are enabled when an engine produces unexpected performance or output.
So I've been reading a lot about "engines lobotomizing the models they run", either because they quantize already quantized weights even further, or because they automatically turn on lossy flags to gain decode/prefill performance. I wanted to test this out by myself, not only with KLD but with real tasks, so I set up an experiment using Qwen 3.8 Flash Next with Basalt (disclaimer: I'm the developer) and two of the most common lossy flags:
- Routing bias: the engine pushes the model toward experts that are already on the GPU, so it sometimes gets a convenient expert instead of the one it actually wanted. - Tail skip: when one of the model's lower expert picks isn't on the GPU, the engine just skips it instead of fetching it.
Both are faster because fewer experts have to come over from RAM, but both change the model's output. I added them to the engine as opt in flags so I could measure exactly what they cost with everything else fixed, only changing the flags.
DISCLAIMER: This post and the experiment were run by me, but it has been polished by AI.