Halcyon paper claims strong text-to-image generation without language pretraining
An arXiv paper posted October 19, "Text-to-Image Generative Modeling Without Language Pretraining," by Keshigeyan Chandrasegaran, Li Fei-Fei, Michael Poli and colleagues, asks whether language pretraining is necessary for strong text-to-image generation. The authors introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100 million image-text pairs. They report that Halcyon-0.52B beats LlamaGen-XL/8 on gFD (48.0 vs. 63.3) with 18.2 times less pretraining compute, and that Halcyon-1.6B beats JiT-T2I-XXL/8++ (35.3 vs. 41.0) with 275 times less compute; controlled experiments found text-only pretraining did not improve T2I generation, and code and models are released.
The paper concludes that strong text-to-image systems can be trained from scratch without internet-scale language pretraining, a claim that, if it holds, challenges the prevailing language-pretraining-first approach.
Computer Science > Machine Learning
arXiv:2610.26863 (cs)
[Submitted on 19 Oct 2026]
Title:Text-to-Image Generative Modeling Without Language Pretraining
Authors:Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas, Ehsan Adeli, Jiajun Wu, Juan Carlos Niebles, Li Fei-Fei, Michael Poli
View a PDF of the paper titled Text-to-Image Generative Modeling Without Language Pretraining, by Keshigeyan Chandrasegaran and 7 other authors