Blockway's Agens Engine keeps a 32B model's 262K context within a 16GB GPU budget
Blockway, a small team from Hong Kong, says 54 of the 72 layers in its 32B Volundr model keep a small fixed-size state instead of a KV cache that grows with every token, and that it built its own MIT-licensed Agens Engine so long context can spill to system RAM within a chosen GPU memory budget. In an uncut demo, a roughly 110K-token novel reads cold in about 69 seconds and a follow-up is answered in 0.5 seconds, with GPU memory peaking around 15.6 GiB; the team discloses that the run was recorded on a 48GB card with the engine held to a 16 GiB plan because it does not yet own a real 16GB card. Blockway also cites Hugging Face user mmssix, who measured 86.4–106.7 tok/s on a 24GB card, and lists an MoE variant, same-GPU image generation, Windows and macOS support, and a compact desktop as future work.
The post targets a concrete local-inference bottleneck, KV-cache growth during long sessions, by replacing most layers' growing cache with fixed-size state and spilling long context to system memory; however, the demo was throttled on a 48GB card rather than run on real 16GB hardware.
| Hey r/LocalLLaMA, Blockway here, a small team from Hong Kong. After our v1 post, the most common question was basically "ok, but why does this model exist?" Totally fair. Here's our honest answer. The problem Long agent sessions eat VRAM. On a normal dense model every layer keeps a KV cache, and it grows with every token, so sooner or later you either cut the context or move to the cloud. We wanted long sessions to stay on your own GPU. So we built Volundr so that 54 of its 72 layers keep a small, fixed-size state instead of a growing cache. Why we wrote our own engine Existing servers aren't really built for that kind of model, so we wrote Agens Engine (MIT) around it. Two things it does that we're proud of: The video One uncut take, real time. A 110K-token novel takes ~69 s to read cold. The follow-up question is answered in 0.5 s, because the engine just picks up where it left off. GPU memory peaks around 15.6 GiB. To be upfront: we recorded it on a 48 GB card with the engine held to a 16 GiB plan. We don't own a real 16 GB card yet. The best real-world numbers so far came from you. mmssix on HF tried it on a 24 GB card we'd never tested and got 86.4–106.7 tok/s. That's in the README now. Thank you! What's next We read every comment last time, and you were right about a lot: We're a small team and there's still a lot to improve. If you give it a try, especially on a real 16 GB card, please tell us honestly how it goes. Every bit of feedback goes into training our next version. What we're hoping for is simple: a world where tokens are free, running on hardware you own. Engine: https://github.com/BlockWayz/agens-engine Weights: https://huggingface.co/collections/Blockway/agens-volundr-32b-v1 submitted by /u/ComfortableKindly507 [link] [comments] |