门控机制如何影响状态空间模型的学习
William L. Tong及合著者报告称,状态空间模型以恒定内存和线性计算进行序列建模,但在上下文学习和精确检索任务上仍落后于Transformer。理论与实验显示,门控会使状态空间模型先学会将信息记入参数,从而延缓甚至阻碍其收敛到正确的上下文学习解,即使架构或内存容量本身并不构成根本限制。研究者还发现,门控通常有助于改善模型对更长序列的泛化能力。
该论文指出,门控是塑造状态空间模型训练动态和泛化能力的关键因素。报告称,这一机制加剧了上下文学习差距,拖慢了状态空间模型在大规模语言建模中的采用,同时门控往往也能改善长序列泛化。
content type paperpublished November 2026
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
AuthorsWilliam L. Tong†**, Aryo Lotfi, Emmanuel Abbe, Kostas Vaggelakos, Vishnu Banna, Etai Littwin, Josh Susskind, Cengiz Pehlevan†, Eran Malach
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context learning and precise retrieval, slowing their adoption for large-scale language modeling. In this work, we demonstrate that both the success and failure of SSMs in these domains can be explained by studying the role of the gating mechanism, a prevalent component in modern recurrent networks. Specifically, we show through theory and experiments that this gating mechanism causes SSMs to first learn an in-weights “memorization” solution, while delaying, or even preventing, convergence to a correct in-context learning solution. Importantly, this happens even in cases where there are no fundamental limitations due to the architecture or its memory capacity. On the other hand, we find that gating is often beneficial for improving generalization to long sequence lengths. Our results illuminate the crucial role of the gating mechanism in shaping both the training dynamics and generalization of SSMs, and provide a basis for understanding and improving linear-time models.