Gated DeltaNet
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 7
- Ai2 launches StateBench for long-context state tracking
Ai2 introduced StateBench, a benchmark with data and code for evaluating how language models track changing entity information across long documents. Its four difficulty levels test increasingly complex state updates; Ai2 reports that directly trained recurrent and hybrid models generalized farther than transformers, while Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna still failed on harder tasks as inputs grew longer.
Ai2 Blog · 🔥 0 - Near-Zero Embeddings Reportedly Stall Gated-Model Training
A paper by Xin Liu and five coauthors surveys 291 open-weight language-model repositories, finding that 52.6% of untied gated and hybrid checkpoints contain reachable input embeddings with norms below 10^-6, compared with 0% of weight-tied models. The authors report that, in gated sequence models such as Mamba, Gated DeltaNet and xLSTM, near-zero embeddings can amplify input gradients, cause FP32 clipping to overflow to infinity and discard a batch’s gradients. During Telugu continual pretraining of Nemotron-3-Nano-4B, the paper reports discarded updates on approximately 20% of steps. The authors propose GroupClip for released checkpoints and GateFloor for pretraining, reporting recovered updates, reduced near-zero gradients and comparable perplexity in their experiments.
Hugging Face · Papers · 🔥 0 - CPU Reproduction Explores Kimi K3’s Kimi Delta Attention
A Juejin AI article presents the first part of a lightweight, CPU-only reproduction of Kimi K3’s Kimi Delta Attention, without GPU use or large-scale training. It analyzes state recurrence, delta-rule writes, bounded logarithmic decay, channel-wise forgetting, head-wise RMSNorm, output gating, and comparisons with other attention and state-space mechanisms, alongside runnable implementation code and demos.
掘金 · 人工智能 · 🔥 0 - Study compares hybrid attention designs for long-context language models
机器之心 reports that a Fudan University and Shanghai Innovation Institute team led by Qiu Xipeng compared RoPE/NoPE full attention, sliding-window attention, and gated linear attention hybrids on long-context tasks. Across models from 376M to 3B parameters, the study found trade-offs between length extrapolation and continued long-context training, and proposed sliding-window linear attention to improve extrapolation. The researchers also analyzed how position-biased and position-free attention modules cooperate.
机器之心 · 🔥 0 - nesformer Compiles an NES CPU Into Qwen3.5 Weights
The nesformer project uses a compiler to produce a standard Qwen3_5ForCausalLM checkpoint containing an emulated NES 6502 CPU, 2 KB of work RAM and cartridge program ROM, rather than training the model on those weights. During play, the model predicts the CPU execution log one token at a time, while a host program handles rendering, controller input and vblank interrupts. The project reports frame-level agreement with a conventional 6502 emulator for its bundled tests, but execution takes roughly 1–6 seconds per frame, and the PPU and APU remain outside the weights.
Hacker News 首页（100+ 分） · 🔥 0 - Developer adds an in-model RISC-V processor to Qwen3.5-4B
Developer Divyansh Agrawal says Qwenputer compiles a deterministic RV32IM processor into Qwen3.5-4B, using Gated DeltaNet state as memory to perform arithmetic and run C and MicroPython code during model forward passes. In developer-reported tests, Qwenputer scored 97.5% on large-integer multiplication, compared with 55% for the base model. IT之家 reports that the model is about 18.8GB and that some operations may take several seconds. Model weights and code are available on Hugging Face, ModelScope and GitHub.
Hacker News 首页（100+ 分） · 🔥 0 - KronLSTM applies the Kronecker Delta Rule to long-context linear attention
An arXiv paper posted on 19 October, "Kronecker Delta Rule for Long Context Linear Attention" by Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer, Sebastian Böck and Sepp Hochreiter, proposes KronLSTM, a linear attention architecture built on the Kronecker Delta Rule, described as a new update rule for associative matrix memories. The authors say KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations by modifying memory along a single direction in matrix space, reducing interference with stored information; they add that memory dynamics can be described as a Markov chain converging to a steady state, with the expected memory-matrix norm uniformly bounded without extra normalization. The paper reports KronLSTM is competitive with, and often outperforms, existing linear attention architectures including the (Gated) DeltaNet family, especially on long-context benchmarks, and says code and models are open-source.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my Gated DeltaNet experienceAsk about Gated DeltaNet
Nobody has shared their experience with Gated DeltaNet yet.