Gated Linear Attention

Headlines · 3

1. Nov 2
   CPU Reproduction Explores Kimi K3’s Kimi Delta Attention

   A Juejin AI article presents the first part of a lightweight, CPU-only reproduction of Kimi K3’s Kimi Delta Attention, without GPU use or large-scale training. It analyzes state recurrence, delta-rule writes, bounded logarithmic decay, channel-wise forgetting, head-wise RMSNorm, output gating, and comparisons with other attention and state-space mechanisms, alongside runnable implementation code and demos.

   掘金 · 人工智能 · 🔥 0
2. Nov 1
   Study compares hybrid attention designs for long-context language models

   机器之心 reports that a Fudan University and Shanghai Innovation Institute team led by Qiu Xipeng compared RoPE/NoPE full attention, sliding-window attention, and gated linear attention hybrids on long-context tasks. Across models from 376M to 3B parameters, the study found trade-offs between length extrapolation and continued long-context training, and proposed sliding-window linear attention to improve extrapolation. The researchers also analyzed how position-biased and position-free attention modules cooperate.

   机器之心 · 🔥 0
3. Oct 20
   Laplace 3B brings fixed-state linear attention to AWS Trainium2

   AWS Machine Learning Blog reports that i14 and AWS adapted the 3B-parameter Laplace 3B model to AWS Trainium2 using native PyTorch through TorchNeuron Native, preserving its Gated Linear Attention architecture and Muon optimizer. On one trn2.48xlarge instance, the implementation trained at about 120,000 tokens per second and served 11,544 output tokens per second across 1,280 concurrent sequences. The blog says the recurrent state remains 29.5 MB per sequence regardless of context length, while warning that fixed-state compression requires workload-specific retrieval-quality evaluation.

   AWS Machine Learning Blog · 🔥 0

Experience and discussion from the community

Share my Gated Linear Attention experienceAsk about Gated Linear Attention