Meta FAIR与纽约大学拆解多模态视觉编码
AI 总结
Meta FAIR与纽约大学研究人员在相同数据和算力条件下训练了200多个多模态语言模型,分别检验视觉编码器预训练、视觉专用参数及其位置的影响。报告称,独立视觉编码器预训练并非必要,但视觉专用参数仍然重要;缺少这些参数的模型在0.6B至6.4B规模下均表现落后。将视觉分支嵌入大语言模型前八层后,VQA表现可与独立编码器相当,首个词元生成时间还低约13%。
深度见解
这些受控比较表明,多模态架构的关键可能是专用的视觉处理能力,而非独立预训练的编码器,可为设计更高效的系统提供参考。
| X threads (with videos!): Paper: https://www.alphaxiv.org/abs/2611.17334 Blog and FAQ: https://unbundling.github.io Labs disagree on vision encoders right now. Gemma 4 12B dropped the encoder, Kimi K3 trains its encoder from scratch, and Inkling 2 Small brought an encoder back. Published models can't settle this, because they differ in data and compute and rarely account for encoder pretraining. A standard encoder also bundles three choices: separate pretraining, parameters that only vision tokens use, and placement before the LLM. We tested each choice separately. Setup: Findings: So vision needs its own parameters, but not its own pretraining or its own encoder. Disclosure: I'm one of the authors. Happy to answer questions! submitted by /u/trungh98 [link] [comments] |
#Meta FAIR#New York University#Qwen3#Gemma 4 12B#Kimi K3#Inkling 2 Small#MetaCLIP