Two 256GB M5U systems reportedly cluster worse than running separate models
A Reddit user reports that clustering two 256GB M5U systems produced worse results than running two models separately, citing concurrency issues and inconsistent decode-speed gains. The poster currently runs Qwen Flash Next q6 and GLM3.5 Flash q4 locally, while using DeepSeek through an API, and says GLM q4 outperformed the API for their coding work.
The account offers an anecdotal practical signal for people evaluating large-memory local LLM hardware, although no controlled benchmarks are provided.
Just an update for anyone looking to get the m5u and thinking about clustering.
I tried clustering but I think running 2 separate models gets significantly better results. The current clustering technology (tensorfold/exos, etc) just isn’t there yet. Especially due to concurrency and the fact that decode doesn’t improve consistently as much as you would hope.
Currently I’m running Qwen flash next q6 and GLM3.5 flash q4. I also have Deepseek on API (from Deepseek). I was previously running 2x Qwen flash but decided to give GLM a shot. GLM is very smart. I’ve found that even q4 is better than my Deepseek API for coding. Qwen is great as the face because it is fast.
I actually think GLM5.3 is slept on. The reports it produces are 100x better than Qwen and Deepseek.
One problem is that 256gb isn’t enough for GLM if you want 500k+ context. I’m stuck at 260k for 3 agents max. And setting it up is finicky. I set my hot cache to 32gb and it caused me to run out of ram which makes speeds drop 70%. So there definitely is some value in the 512gb version of the m5u (hopefully newer version make the kv cache cheaper on RAM).
If anyone has any better optimizations or ways to run things on the 2 256gb m5u I’m all ears.
submitted by /u/Viktri1 [link] [comments]