Cortex distills context gains into parametric memory
A technical report on arXiv, "Cortex: Consolidating Context into Parametric Memory," was submitted on 14 October 2026 by Jintian Zhang, Feng He, Xinnian Liang, Ningyu Zhang and four other authors. The paper frames Memory Consolidation as internalizing context-induced gains while preserving non-target capabilities, and introduces a five-task-family testbed that measures consolidation and retention under single, batch and sequential updates. Cortex generates prospective queries to anticipate how context will be used, then distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. The authors report that across 4B–120B dense and mixture-of-experts backbones, including text-only and multimodal models, Cortex sits on the empirical consolidation–retention Pareto frontier in most evaluated settings; after 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance. These figures are author-reported and no independent replication is provided.
The report notes that context can improve large language model performance but those gains may be lost after context removal, which is the problem the consolidation objective is meant to address.
Computer Science > Computation and Language
arXiv:2610.20399 (cs)
[Submitted on 14 Oct 2026]
Title:Cortex: Consolidating Context into Parametric Memory
Authors:Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang
View a PDF of the paper titled Cortex: Consolidating Context into Parametric Memory, by Jintian Zhang and 7 other authors