Meta FAIR and NYU study untangles multimodal vision encoding
Meta FAIR and New York University researchers trained more than 200 multimodal language models under matched data and compute conditions to separate the effects of encoder pretraining, vision-specific parameters, and placement. They report that separate vision encoder pretraining was optional, while vision-only parameters remained important; models without them trailed across sizes from 0.6B to 6.4B. Embedding vision-only branches in the first eight LLM layers matched a separate encoder on VQA while achieving about 13% lower time to first token.
The controlled comparisons indicate that multimodal architectures may rely more on dedicated vision capacity than on a separately pretrained encoder, helping researchers design more efficient systems.
| X threads (with videos!): Paper: https://www.alphaxiv.org/abs/2611.17334 Blog and FAQ: https://unbundling.github.io Labs disagree on vision encoders right now. Gemma 4 12B dropped the encoder, Kimi K3 trains its encoder from scratch, and Inkling 2 Small brought an encoder back. Published models can't settle this, because they differ in data and compute and rarely account for encoder pretraining. A standard encoder also bundles three choices: separate pretraining, parameters that only vision tokens use, and placement before the LLM. We tested each choice separately. Setup: Findings: So vision needs its own parameters, but not its own pretraining or its own encoder. Disclosure: I'm one of the authors. Happy to answer questions! submitted by /u/trungh98 [link] [comments] |
#Meta FAIR#New York University#Qwen3#Gemma 4 12B#Kimi K3#Inkling 2 Small#MetaCLIP