Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Qwen3.8-Flash-Next 125B benchmarked on 24GB Apple Silicon Macs

AI summary

A Reddit contributor benchmarked the 76GB-on-disk Qwen3.8-Flash-Next 125B GSQ-RCO IQ3_XXS quantization with the MoEShift backend on 24GB M4 Pro and M5 Pro Macs. Median results across three runs with prompt caching disabled included 374.15 tok/s prefill and 14.69 tok/s short-prompt decode on the M5 Pro, with peak physical process memory of 8.5GB on both systems.

In-depth insight

The community measurements show that a 125B mixture-of-experts model can run with useful throughput and an 8.5GB process footprint on 24GB Apple Silicon Macs by offloading weights to SSD.

I benchmarked Qwen3.8-Flash-Next 125B ISTA-DASLab's GSQ-RCO IQ3_XXS quant (76GB on disk), on two Macs with 24GB unified memory using MoEShift backend:

MetricM4 Pro (medians)M5 Pro (medians)
Short prompt decode (72 input tokens)9.84 tok/s14.69 tok/s
Long prompt prefill (6,060 input tokens)199.09 tok/s374.15 tok/s
Long prompt decode (512 output tokens)9.57 tok/s13.14 tok/s
Long prompt inference time83.9s55.1s
Peak physical process memory8.5GB8.5GB

These are medians of 3 runs using the MoEShift community benchmark runner (prompt caching disabled). The results are particularly interesting for prefill making this model perfectly usable in a chat or in an agent, while both machines stayed within 8.5GB peak process footprint. This is a very promising benchmark even for the upcoming Qwen4 MoE.

#Qwen3.8-Flash-Next 125B#MoEShift#ISTA-DASLab#GSQ-RCO IQ3_XXS#Apple M4 Pro#Apple M5 Pro

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.