Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

RAVEL scores vision encoders for MLLMs without training

AI summary

A research team trained and evaluated 210 multimodal large language models (MLLMs) by pairing 70 vision-encoder configurations with Qwen3-1.7B, Qwen2.5-1.5B and SmolLM2-1.7B under a uniform LLaVA-style two-stage recipe, then testing them on 11 downstream benchmarks, according to a report by Machine Heart. They found that as the candidate pool of vision encoders grows more numerous and diverse, existing cross-modal evaluation metrics lose predictive power and can even fall behind purely unimodal vision-only measures. The team's proposed fix, RAVEL, is training-free: it scores a candidate encoder by how consistent image-side and text-side nearest-neighbor relations are, adding stabilized PCA whitening and patch-level similarity matching. The authors report an average Spearman correlation of 0.85 with real downstream performance across the three LLMs, 36.1% above the trained Alignment Probing method and 93.5% and 101.1% above the training-free GW and MutualNN baselines; code, the paper and the 210 MLLM checkpoints are open-sourced.

In-depth insight

It offers a cheap, training-free way to shortlist vision encoders before committing to full cross-modal alignment training, and releases a reusable model zoo.

机器之心 2026-10-17 12:41 北京

如何在完整训练 MLLM 之前,为目标 LLM 选择合适的视觉编码器?

视觉编码器是多模态大模型(MLLM)的核心组件之一。即使使用相同的大语言模型(LLM),不同的视觉编码器也可能带来显著不同的下游表现。要找到最合适的视觉编码器,最直接的方法是将候选编码器逐一接入 LLM,完成跨模态对齐训练后进行评测,但这一过程需要大量计算资源。因此,如何在完整训练之前准确预测视觉编码器的下游表现,成为一个重要的研究问题。

#RAVEL#Qwen3-1.7B#Qwen2.5-1.5B#SmolLM2-1.7B#LLaVA

Original · 8 min readRead the original
Group chat

No comments yet. Start the conversation.