Cola2 proposes continuous semantic compression for language modeling
The authors of the Cola2 paper propose a hierarchical language-modeling framework that combines continuous semantic compression, reconstruction and generation supervision, and co-adaptation between continuous and discrete representations. The system dynamically segments text into semantic spans and uses auxiliary-loss-free negative feedback to control compression, while a block-causal masked diffusion decoder handles generation. The authors report experiments on five downstream tasks using about 210 billion training tokens, but the performance claims have not been independently verified.
Cola2 offers a research architecture for reducing the gap between continuous semantic representations and discrete text generation, with reported scaling results at large training-token volumes.
Computer Science > Computation and Language
arXiv:2610.31305 (cs)
[Submitted on 22 Oct 2026]
Title:Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation
Authors:Hongcan Guo, Yian Zhao, Feng Wang, Shen Nie, Guoqiang Wei, Yan Zeng, Hengshuang Zhao
View a PDF of the paper titled Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation, by Hongcan Guo and 6 other authors