How Gating Shapes State Space Model Learning
William L. Tong and coauthors report that state space models use constant memory and linear computation for sequence modeling but lag behind Transformers on in-context learning and precise retrieval. Their theory and experiments show that gating causes state space models to first learn an in-weights memorization solution, delaying or preventing convergence to correct in-context learning even when architecture or memory capacity is not the fundamental constraint. The researchers also find that gating often improves generalization to longer sequence lengths.
The paper identifies gating as a crucial factor shaping state space models’ training dynamics and generalization. It reports that this mechanism contributes to the in-context-learning gap that slows adoption of state space models for large-scale language modeling, while also often benefiting long-sequence generalization.
content type paperpublished November 2026
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
AuthorsWilliam L. Tong†**, Aryo Lotfi, Emmanuel Abbe, Kostas Vaggelakos, Vishnu Banna, Etai Littwin, Josh Susskind, Cengiz Pehlevan†, Eran Malach
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context learning and precise retrieval, slowing their adoption for large-scale language modeling. In this work, we demonstrate that both the success and failure of SSMs in these domains can be explained by studying the role of the gating mechanism, a prevalent component in modern recurrent networks. Specifically, we show through theory and experiments that this gating mechanism causes SSMs to first learn an in-weights “memorization” solution, while delaying, or even preventing, convergence to a correct in-context learning solution. Importantly, this happens even in cases where there are no fundamental limitations due to the architecture or its memory capacity. On the other hand, we find that gating is often beneficial for improving generalization to long sequence lengths. Our results illuminate the crucial role of the gating mechanism in shaping both the training dynamics and generalization of SSMs, and provide a basis for understanding and improving linear-time models.