Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Sparse-KV extension runs eight 250K-context coding agents on two RTX 3090s

AI summary

A Reddit author reports a sparse-attention extension for HyperQwen that keeps full int4 KV caches in system RAM while retrieving roughly 1% of tokens through a rank-64 key sketch. On a 3090 Ti and 3090, the author measured eight concurrent Qwen3.8-27B-W4A16 coding agents with 250K-token contexts at about 545 tokens per second on explain-style output, 31/32 recall, and roughly five-second follow-up starts. The author also reports that approximate attention can occasionally confuse near-duplicate code lines.

In-depth insight

The published patches and experiments offer a concrete way to reduce repeated KV-cache transfers while preserving long contexts for multiple concurrent coding agents, though the reported figures remain self-measured.

I am wide open to the idea that I am misguided or misinformed, but my findings feel worth sharing:

Short video of my UI with 8 agents each explaining a codebase in parallel long prose: https://imgur.com/JSqlXIi

A few days ago I read the Stanford/Berkeley TokenGate paper. The idea is to make long context cheap by pruning tokens up front.

I tried it. It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard.

So I went the other way - don't throw anything away, just don't look at all of it at once. Keep the whole context in system RAM and at each step pull in only the ~1% that matters right now.

A few days later, I have two 3090s run 8 coding agents at once, each holding their own 250K-token context:

#SparseKV-HyperQwen#Qwen3.8-27B-W4A16-AutoRound#HyperQwen#vLLM#Triton#NVIDIA RTX 3090#NVIDIA RTX 3090 Ti

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.