Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Reddit user runs Qwen Flash Next at 24 tokens per second on Strata

AI summary

A Reddit user reports running Qwen Flash Next locally through Strata at 24 tokens per second on a 64GB M1 Max MacBook Pro. They say the model performs well for basic tasks but does not compete with larger frontier models. The report also mentions MoEShift and offloading larger models to SSD storage.

In-depth insight

The account is an early deployment signal that Strata and SSD offloading may expand local-model options on consumer hardware, though it is based on one user's experience rather than controlled benchmarks.

I have been looking for a local model to do some of the typical work I used AI subscriptions for.

I’ve tested almost every popular model. I have an M1 Max 64gb MacBook Pro as the inference machine and because of the memory size, I wasn’t able to load anything very effective in the tasks I use it for.

Found out about strata and have been testing it out. It’s awesome! I’m kinda surprised more people don’t talk about strata on here. I feel like a lot of us are in that limbo area where we have hardware that is more powerful than needed to run a sub 30b model like Qwen 27b, but not large enough to run models like Qwen Flash Next.

But thanks to open source research, I’m now getting Qwen Flash Next to run at 24 tok/s! And it’s doing phenomenal work! I know it’s not going to compete with the larger frontier models, but I don’t think I necessarily “need” frontier model intelligence. I just need enough to do basic things while I can handle the rest.

Seriously, check out Strata and also MoEShift that someone posted about on here. It’s awesome to be able to offload larger models onto SSD. I hope this offloading technique grows and more research is done on it because I genuinely think this could be an amazing way to get these frontier level open source models to work on regular consumer hardware.

#Qwen Flash Next#Strata#MoEShift#Qwen 27B#M1 Max#MacBook Pro

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.