TaSQ

Headlines · 1

1. Nov 3
   LoopVQ Targets Sub-1-Bit KV Cache Quantization

   Minsoo Cheong and Sungjoo Yoo’s paper introduces LoopVQ, a KV cache vector quantization method for looped transformers that jointly codes channels across multiple loop steps and fine-tunes codebooks against full-precision outputs. The authors report that it outperforms existing vector quantization methods on general, long chain-of-thought reasoning and long-context retrieval tasks with Ouro and Huginn at 0.5 bits per element. In SGLang, the implementation nearly halves decode latency at a 32K-token context and raises peak serving throughput by up to 4.3× and 6.1× for the 1.4B and 2.6B Ouro models versus BF16.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my TaSQ experienceAsk about TaSQ