苹果提出加权状态混合检索法
Apple Machine Learning Research发表论文,提出加权状态混合方法,通过学习与查询相关的权重来组合缓存的状态空间模型表示。作者称,该方法在文档问答、检索、变量追踪和工具使用等长上下文任务上取得显著提升;其稀疏变体的编码成本相对于上下文长度接近恒定。该论文被NeurIPS 2026的PALM研讨会接收,但性能声明尚无独立佐证。
该方法将可复用的缓存上下文与选择性检索结合,旨在应对长而拥挤的上下文给状态空间模型带来的效率与可靠性问题。
content type paperpublished November 2026
Weighted State Souping for Efficient Long-Context Retrieval
AuthorsWilliam L. Tong†**, Aryo Lotfi, Emmanuel Abbe, Josh Susskind, Cengiz Pehlevan†, Eran Malach
This paper was accepted at the Workshop on Personalized, Aligned, Long-term Memory for AI Systems (PALM) at NeurIPS 2026.
State space models (SSMs) combine fast inference and fixed state size with competitive language-modeling performance. Recently, state souping has been proposed as a procedure whereby an SSM independently encodes chunks of tokens like documents, tool definitions, or memory records. The resulting state encodings may be cached and recombined to cheaply construct a context for downstream queries. However, both state souping and vanilla SSMs deteriorate on long, crowded contexts, limiting their utility for long-horizon domains. To address this issue, we propose weighted state souping, a simple method that learns query-dependent weights for composing cached states. The resulting models improve long-context performance substantially across both single- and multi-hop document question answering, needle-in-a-haystack retrieval, variable tracking, and tool use. Our method can be further adapted into an efficient sparse variant, yielding near-constant encoding cost with respect to context length while preserving (and sometimes improving) performance. We also present simple theoretical arguments that illustrate why souping works and when it may fail, and discuss informally the advantages of weighted souping. Weighted state souping thus combines reusable context representations with selective retrieval, offering an efficient approach to language modeling over large collections of documents, tools, and other forms of structured context that increasingly dominate modern agentic workloads.
#Apple Machine Learning Research#Weighted State Souping#NeurIPS 2026