创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

64GB M4 Max运行27B长上下文编程智能体

AI 总结

一名Reddit用户称,其在64GB M4 Max上通过LM Studio运行Qwen3.8 27B,处理约10万至20万token的编程智能体会话。上下文达到16.4万token时,LM Studio引擎占用48GB内存,其中46GB为GPU内存;缓存未命中时,完整预填充约需20分钟。作者表示提示缓存使该配置得以实用,但LM Studio的MLX视觉路径及混合模型特性限制了KV缓存量化。

深度见解

这份第一人称实测提供了长上下文本地智能体配置的内存和预填充数据,对考虑使用64GB Mac的用户有参考价值,但尚未得到复现验证。

https://preview.redd.it/hboxdgjh2xyh1.png?width=1230&format=png&auto=webp&s=cca6b2b7b08807a334ff5280e19d96133667fbb0 The usual advice was that 64GB is a dead zone for local LLMs: too small for a 70B with real context, and more than a 30B needs. After running a 27B for agentic coding on a 64GB M4 Max for the last ten days, I think that has flipped. My memory usage is in the 50 GB as the context starts building up. Setup: Qwen3.8 27B (16 GB MLX quant) in LM Studio, 256k context, coding agent sessions that sit at 100-200k tokens. At 164k tokens, LM Studio's engine process is using 48 GB, 46 GB of it GPU memory. Roughly: - 16 GB weights - ~11 GB KV cache (~64 KB/token, only 16 of the 64 layers are full attention) - what looks like a second copy of the context for the prompt cache, plus the linear attention checkpoints - MLX holding on to scratch memory from prefill The prompt cache is what makes it usable. It's hitting 93%, so a turn processes about 1k new tokens instead of 160k. When it misses, a full prefill takes about 20 minutes. Why not quantize the KV cache like people do on a 24GB card? LM Studio's MLX vision path refuses KV quantization for this model. The text-only path allows it, but skips the cache checkpoints when the cache is quantized, and on a hybrid model that means a lot more full reprocessing. So on 48GB this doesn't fit unless you drop to ~128k context or move to llama.cpp with q8 KV (should be ~25-30 GB at 256k, with slower prefill). On 64GB it fits with nothing to spare. Free memory is about 150 MB right now. The dead zone argument was about weights. With agents, context costs about as much as the model, and 27B models are now good enough for real work. That puts 64GB right in the sweet spot. Bigger MoE models or two models loaded at once still want 128GB. Anyone else on 64GB seeing the same? And is anyone running long context on llama.cpp on a Mac with prefill speeds they're happy with? submitted by /u/M5_Maxxx [link] [comments]

#Qwen3.8 27B#LM Studio#Apple M4 Max#llama.cpp

原文约 2 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。