Basalt 开发者测试有损 MoE 路由选项
一名 Basalt 开发者报告了使用 Qwen 3.8 Flash Next 进行的实验，测试了两个默认需手动启用的有损推理选项：路由偏置会优先使用已驻留 GPU 的专家，尾部跳过则省略未驻留 GPU 的较低排名专家。两者都能减少从内存传入专家的次数，但可能改变输出。在模拟小显存显卡的配置中，随附设置使文本吞吐量提高 56%，同时阶梯任务和事实任务得分分别下降 5.2 分和 2.3 分。开发者刻意设置的最差参数使事实正确率从 38.1% 降至 2.7%。测试仅覆盖一套硬件上的一个模型；阶梯任务使用一个种子，事实任务使用三个种子，吞吐量来自单次冷启动运行。Basalt 默认关闭这两个选项，并在启用时发出警告。
报告表明，以速度为导向的 MoE 推理选项可能改变事实记忆和链式推理表现，尤其是在模型无法完全装入显存时。因此，当引擎出现异常性能或输出时，报告支持检查这些选项是否已启用。
So I've been reading a lot about "engines lobotomizing the models they run", either because they quantize already quantized weights even further, or because they automatically turn on lossy flags to gain decode/prefill performance. I wanted to test this out by myself, not only with KLD but with real tasks, so I set up an experiment using Qwen 3.8 Flash Next with Basalt (disclaimer: I'm the developer) and two of the most common lossy flags:
- Routing bias: the engine pushes the model toward experts that are already on the GPU, so it sometimes gets a convenient expert instead of the one it actually wanted. - Tail skip: when one of the model's lower expert picks isn't on the GPU, the engine just skips it instead of fetching it.
Both are faster because fewer experts have to come over from RAM, but both change the model's output. I added them to the engine as opt in flags so I could measure exactly what they cost with everything else fixed, only changing the flags.
DISCLAIMER: This post and the experiment were run by me, but it has been polished by AI.