TokenGate

Headlines · 1

1. Oct 21
   TokenGate adds learnable token pruning to speed long-context prefill

   Researchers posted a paper on r/LocalLLaMA describing TokenGate, a small drop-in component that extends already-trained models to enable end-to-end learnable token pruning over long-context inputs for faster and more memory-efficient prefill. To learn despite the non-differentiable top-K operator, the authors introduce token-gated attention, a Mixture-of-Experts-inspired mechanism that re-weights attention logits using token importance scores. They report 1.4–11.3× faster time-to-first-token and 1.5–3.4× lower peak memory with no accuracy degradation for prefill, and up to 6.4× faster prefill and 2× lower peak memory than the best prior token-pruning methods, all figures self-reported. The authors say TokenGate natively extends standard transformers such as Qwen3 and hybrid-attention models such as Qwen3.5 with lightweight adaptation, and they released code on GitHub.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my TokenGate experienceAsk about TokenGate