Blockway自研引擎让32B模型在16GB显存跑长上下文
AI 总结
香港小型团队 Blockway 称，其 32B 模型 Volundr 的 72 层中有 54 层采用固定大小状态，而不是随 token 增长的 KV 缓存，并为此自研 MIT 许可的 Agens Engine，让长上下文可按设定的显存预算溢出到系统内存。团队发布一段未剪辑演示：约 11 万 token 的小说冷读约 69 秒，追问 0.5 秒完成，显存峰值约 15.6 GiB；但他们说明视频是在 48GB 显卡上把引擎限制在 16 GiB 方案录制的，团队尚无真正的 16GB 显卡。Blockway 还称 Hugging Face 用户 mmssix 在 24GB 卡上测得 86.4–106.7 tok/s，MoE 变体、同卡图像生成以及 Windows/macOS 支持仍在路线图上。
深度见解
这篇帖子针对本地推理中长会话 KV 缓存持续增长这一具体瓶颈，用固定大小状态替代多数层的增长缓存，并把长上下文溢出到系统内存；但演示是在 48GB 卡上限速录制的。
| Hey r/LocalLLaMA, Blockway here, a small team from Hong Kong. After our v1 post, the most common question was basically "ok, but why does this model exist?" Totally fair. Here's our honest answer. The problem Long agent sessions eat VRAM. On a normal dense model every layer keeps a KV cache, and it grows with every token, so sooner or later you either cut the context or move to the cloud. We wanted long sessions to stay on your own GPU. So we built Volundr so that 54 of its 72 layers keep a small, fixed-size state instead of a growing cache. Why we wrote our own engine Existing servers aren't really built for that kind of model, so we wrote Agens Engine (MIT) around it. Two things it does that we're proud of: The video One uncut take, real time. A 110K-token novel takes ~69 s to read cold. The follow-up question is answered in 0.5 s, because the engine just picks up where it left off. GPU memory peaks around 15.6 GiB. To be upfront: we recorded it on a 48 GB card with the engine held to a 16 GiB plan. We don't own a real 16 GB card yet. The best real-world numbers so far came from you. mmssix on HF tried it on a 24 GB card we'd never tested and got 86.4–106.7 tok/s. That's in the README now. Thank you! What's next We read every comment last time, and you were right about a lot: We're a small team and there's still a lot to improve. If you give it a try, especially on a real 16 GB card, please tell us honestly how it goes. Every bit of feedback goes into training our next version. What we're hoping for is simple: a world where tokens are free, running on hardware you own. Engine: https://github.com/BlockWayz/agens-engine Weights: https://huggingface.co/collections/Blockway/agens-volundr-32b-v1 submitted by /u/ComfortableKindly507 [link] [comments] |