Qwen-Flash-Next benchmarked on a 16GB M2 Pro Mac Mini with Strata
A user loaded Strata and ran the iQ3_xxs quantization of Qwen-Flash-Next on a 16GB M2 Pro Mac mini, setting a maximum context of 128k, and used opencode on a separate workstation to build a script benchmarking performance and output quality on typical tasks such as large CSV sorting and low-end code work at varying context levels. The author reported it worked far better than expected. The author assumes speeds were always at their worst because nothing could be cached and experts were repeatedly re-read and swapped, which would mean prefill and decode token speed stay the same regardless of load. Median speed across tests from 16k to 120k context was 77 tok/s prefill and 5.5 tok/s decode. The author judged it barely passable as a chat generator but not hopeless as a last-resort "living on dial-up" code helper, and said it appears pretty rough on the SSD, though the write load is not much worse than using a Mini as an active DB server or other office server task.
I was getting ready to sell an old kiosk Mini at our office that was getting an upgrade, and figured I would see what happened if I tried to make it do something it by all rights shouldn't be able to. I loaded up Strata and ran the iQ3_xxs quant of QFN with a max of 128k context. I used opencode on my normal workstation to build a script that would benchmark the performance and output quality of typical tasks I do (mostly large csv sorting and low end code grunt work)at varying levels of context.
I have to say, it worked far better than I would have expected it to. I assume that the speed is always at its worst (as if it had full context) because it never has a chance to cache anything and is always re-reading and swapping experts across prompts, but as an aside it means that token speed for both prefill and decode are the same regardless how much you're making it do. The median speed between all tests from 16k to 120k context was 77 tok/s prefill and 5.5 tok/s decode. That means it could actually stand in as a barely passable chat generator, and is not hopeless as a "living on dial-up" code helper of last resort. It does appear that it is pretty rough on the SSD, but the write load is not much worse than someone using a Mini as a pretty active DB server or other "proper" server task in an operating office environment.