xLSTM

Headlines · 2

1. Nov 2
   Near-Zero Embeddings Reportedly Stall Gated-Model Training

   A paper by Xin Liu and five coauthors surveys 291 open-weight language-model repositories, finding that 52.6% of untied gated and hybrid checkpoints contain reachable input embeddings with norms below 10^-6, compared with 0% of weight-tied models. The authors report that, in gated sequence models such as Mamba, Gated DeltaNet and xLSTM, near-zero embeddings can amplify input gradients, cause FP32 clipping to overflow to infinity and discard a batch’s gradients. During Telugu continual pretraining of Nemotron-3-Nano-4B, the paper reports discarded updates on approximately 20% of steps. The authors propose GroupClip for released checkpoints and GateFloor for pretraining, reporting recovered updates, reduced near-zero gradients and comparable perplexity in their experiments.

   Hugging Face · Papers · 🔥 0
2. Oct 18
   KronLSTM applies the Kronecker Delta Rule to long-context linear attention

   An arXiv paper posted on 19 October, "Kronecker Delta Rule for Long Context Linear Attention" by Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer, Sebastian Böck and Sepp Hochreiter, proposes KronLSTM, a linear attention architecture built on the Kronecker Delta Rule, described as a new update rule for associative matrix memories. The authors say KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations by modifying memory along a single direction in matrix space, reducing interference with stored information; they add that memory dynamics can be described as a Markov chain converging to a steady state, with the expected memory-matrix norm uniformly bounded without extra normalization. The paper reports KronLSTM is competitive with, and often outperforms, existing linear attention architectures including the (Gated) DeltaNet family, especially on long-context benchmarks, and says code and models are open-source.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my xLSTM experienceAsk about xLSTM