64GB M4 Max Runs a 27B Model for Long-Context Coding Agents
AI summary
A Reddit user reports running Qwen3.8 27B in LM Studio on a 64GB M4 Max for coding-agent sessions containing roughly 100,000 to 200,000 tokens. At 164,000 tokens, the LM Studio engine reportedly used 48GB of memory, including 46GB of GPU memory; cache misses triggered a full prefill of about 20 minutes. The author says prompt caching made the setup usable, while quantized KV caching was limited by LM Studio’s MLX vision path and hybrid-model behavior.
In-depth insight
The first-hand account provides concrete memory and prefill measurements for a long-context local agent setup, offering practical data for users considering 64GB Macs while remaining unverified.
| https://preview.redd.it/hboxdgjh2xyh1.png?width=1230&format=png&auto=webp&s=cca6b2b7b08807a334ff5280e19d96133667fbb0 The usual advice was that 64GB is a dead zone for local LLMs: too small for a 70B with real context, and more than a 30B needs. After running a 27B for agentic coding on a 64GB M4 Max for the last ten days, I think that has flipped. My memory usage is in the 50 GB as the context starts building up. Setup: Qwen3.8 27B (16 GB MLX quant) in LM Studio, 256k context, coding agent sessions that sit at 100-200k tokens. At 164k tokens, LM Studio's engine process is using 48 GB, 46 GB of it GPU memory. Roughly: - 16 GB weights - ~11 GB KV cache (~64 KB/token, only 16 of the 64 layers are full attention) - what looks like a second copy of the context for the prompt cache, plus the linear attention checkpoints - MLX holding on to scratch memory from prefill The prompt cache is what makes it usable. It's hitting 93%, so a turn processes about 1k new tokens instead of 160k. When it misses, a full prefill takes about 20 minutes. Why not quantize the KV cache like people do on a 24GB card? LM Studio's MLX vision path refuses KV quantization for this model. The text-only path allows it, but skips the cache checkpoints when the cache is quantized, and on a hybrid model that means a lot more full reprocessing. So on 48GB this doesn't fit unless you drop to ~128k context or move to llama.cpp with q8 KV (should be ~25-30 GB at 256k, with slower prefill). On 64GB it fits with nothing to spare. Free memory is about 150 MB right now. The dead zone argument was about weights. With agents, context costs about as much as the model, and 27B models are now good enough for real work. That puts 64GB right in the sweet spot. Bigger MoE models or two models loaded at once still want 128GB. Anyone else on 64GB seeing the same? And is anyone running long context on llama.cpp on a Mac with prefill speeds they're happy with? submitted by /u/M5_Maxxx [link] [comments] |