125B模型在24GB苹果芯片Mac上的社区基准
一名Reddit贡献者使用MoEShift后端,在24GB统一内存的M4 Pro和M5 Pro Mac上测试了磁盘占用76GB的Qwen3.8-Flash-Next 125B GSQ-RCO IQ3_XXS量化版。关闭提示缓存的三次运行中位结果显示,M5 Pro预填充速度为374.15 tok/s,短提示解码速度为14.69 tok/s,两台机器的物理进程内存峰值均为8.5GB。
社区实测表明,通过将权重卸载至SSD,125B混合专家模型可在24GB统一内存的苹果芯片Mac上以8.5GB进程内存占用获得可用吞吐量。
I benchmarked Qwen3.8-Flash-Next 125B ISTA-DASLab's GSQ-RCO IQ3_XXS quant (76GB on disk), on two Macs with 24GB unified memory using MoEShift backend:
| Metric | M4 Pro (medians) | M5 Pro (medians) |
|---|---|---|
| Short prompt decode (72 input tokens) | 9.84 tok/s | 14.69 tok/s |
| Long prompt prefill (6,060 input tokens) | 199.09 tok/s | 374.15 tok/s |
| Long prompt decode (512 output tokens) | 9.57 tok/s | 13.14 tok/s |
| Long prompt inference time | 83.9s | 55.1s |
| Peak physical process memory | 8.5GB | 8.5GB |
These are medians of 3 runs using the MoEShift community benchmark runner (prompt caching disabled). The results are particularly interesting for prefill making this model perfectly usable in a chat or in an agent, while both machines stayed within 8.5GB peak process footprint. This is a very promising benchmark even for the upcoming Qwen4 MoE.
#Qwen3.8-Flash-Next 125B#MoEShift#ISTA-DASLab#GSQ-RCO IQ3_XXS#Apple M4 Pro#Apple M5 Pro