稀疏KV扩展在双3090上运行8个长上下文编码智能体
Reddit 作者报告了一套面向 HyperQwen 的稀疏注意力扩展,将完整 int4 KV 缓存保留在系统内存中,并通过 rank-64 键草图每步仅检索约 1% 的 token。其在 3090 Ti 与 3090 上测试了 8 个并发 Qwen3.8-27B-W4A16 编码智能体,每个上下文为 250K,解释型输出总吞吐量约为 545 token/s,召回结果为 31/32,后续对话约 5 秒启动。作者同时指出,近似注意力偶尔会混淆相似的代码行。
作者公开了补丁与实验代码,为减少长上下文 KV 缓存传输并支持多个编码智能体并发提供了一套可检验方案,但相关性能数字仍属于自测。
I am wide open to the idea that I am misguided or misinformed, but my findings feel worth sharing:
Short video of my UI with 8 agents each explaining a codebase in parallel long prose: https://imgur.com/JSqlXIi
A few days ago I read the Stanford/Berkeley TokenGate paper. The idea is to make long context cheap by pruning tokens up front.
I tried it. It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard.
So I went the other way - don't throw anything away, just don't look at all of it at once. Keep the whole context in system RAM and at each step pull in only the ~1% that matters right now.
A few days later, I have two 3090s run 8 coding agents at once, each holding their own 250K-token context:
#SparseKV-HyperQwen#Qwen3.8-27B-W4A16-AutoRound#HyperQwen#vLLM#Triton#NVIDIA RTX 3090#NVIDIA RTX 3090 Ti