Compressed Sparse Attention 2

Headlines · 1

1. Oct 28
   Reddit user proposes architectural priorities for Gemma 5

   A Reddit user shared suggestions for future Gemma models, including a looped transformer, DeepSeek’s Engram, continued 4-bit quantization and more efficient attention mechanisms. The post also proposes a roughly 120B mixture-of-experts model, improved drafter models, linear projections for vision and audio, and abandoning 2-bit quantization for small models. These are community proposals, not a Google announcement; the post contains no Gemma 5 implementation or benchmark results.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my Compressed Sparse Attention 2 experienceAsk about Compressed Sparse Attention 2