RAVEL scores vision encoders for MLLMs without training
A research team trained and evaluated 210 multimodal large language models (MLLMs) by pairing 70 vision-encoder configurations with Qwen3-1.7B, Qwen2.5-1.5B and SmolLM2-1.7B under a uniform LLaVA-style two-stage recipe, then testing them on 11 downstream benchmarks, according to a report by Machine Heart. They found that as the candidate pool of vision encoders grows more numerous and diverse, existing cross-modal evaluation metrics lose predictive power and can even fall behind purely unimodal vision-only measures. The team's proposed fix, RAVEL, is training-free: it scores a candidate encoder by how consistent image-side and text-side nearest-neighbor relations are, adding stabilized PCA whitening and patch-level similarity matching. The authors report an average Spearman correlation of 0.85 with real downstream performance across the three LLMs, 36.1% above the trained Alignment Probing method and 93.5% and 101.1% above the training-free GW and MutualNN baselines; code, the paper and the 210 MLLM checkpoints are open-sourced.
It offers a cheap, training-free way to shortlist vision encoders before committing to full cross-modal alignment training, and releases a reusable model zoo.
机器之心 2026-10-17 12:41 北京
如何在完整训练 MLLM 之前,为目标 LLM 选择合适的视觉编码器?
视觉编码器是多模态大模型(MLLM)的核心组件之一。即使使用相同的大语言模型(LLM),不同的视觉编码器也可能带来显著不同的下游表现。要找到最合适的视觉编码器,最直接的方法是将候选编码器逐一接入 LLM,完成跨模态对齐训练后进行评测,但这一过程需要大量计算资源。因此,如何在完整训练之前准确预测视觉编码器的下游表现,成为一个重要的研究问题。