KronLSTM

Headlines · 1

1. Oct 18
   KronLSTM applies the Kronecker Delta Rule to long-context linear attention

   An arXiv paper posted on 19 October, "Kronecker Delta Rule for Long Context Linear Attention" by Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer, Sebastian Böck and Sepp Hochreiter, proposes KronLSTM, a linear attention architecture built on the Kronecker Delta Rule, described as a new update rule for associative matrix memories. The authors say KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations by modifying memory along a single direction in matrix space, reducing interference with stored information; they add that memory dynamics can be described as a Markov chain converging to a steady state, with the expected memory-matrix norm uniformly bounded without extra normalization. The paper reports KronLSTM is competitive with, and often outperforms, existing linear attention architectures including the (Gated) DeltaNet family, especially on long-context benchmarks, and says code and models are open-source.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my KronLSTM experienceAsk about KronLSTM