DeepSeek Engram

Headlines · 2

1. Oct 28
   Reddit user proposes architectural priorities for Gemma 5

   A Reddit user shared suggestions for future Gemma models, including a looped transformer, DeepSeek’s Engram, continued 4-bit quantization and more efficient attention mechanisms. The post also proposes a roughly 120B mixture-of-experts model, improved drafter models, linear projections for vision and audio, and abandoning 2-bit quantization for small models. These are community proposals, not a Google announcement; the post contains no Gemma 5 implementation or benchmark results.

   r/LocalLLaMA · 🔥 0
2. Oct 6
   EngramEdit edits facts in LLMs through conditional memory

   A new arXiv paper, EngramEdit, builds on conditional memory architectures such as DeepSeek Engram — which look up learned embeddings from input n-grams to expand model capacity at limited extra computation — to decouple factual knowledge storage from general computation and update facts while keeping the Transformer backbone fixed. It first computes target memory representations that make the model predict the updated fact across multiple expressions, then jointly updates the shared n-gram embeddings, penalizing changes to frequently reused embeddings more strongly to preserve unrelated knowledge. The authors report near-perfect editing success, generalization of revised knowledge to unseen expressions and multi-hop reasoning, roughly three times the strongest baseline's chain-of-thought accuracy, and largely preserved unrelated knowledge and general capabilities as edits accumulate.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my DeepSeek Engram experienceAsk about DeepSeek Engram