Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Multimodal Flow Unifies Language and Vision With Continuous Modeling

AI summary

Huazhong University of Science and Technology, Horizon Robotics, and other collaborators introduced Multimodal Flow, a fully continuous architecture that models language and vision in their native embedding spaces under one Flow Matching objective. Its 1.6B-parameter MF-1 model was pretrained from scratch on 150B tokens and, according to the report, outperformed Transfusion-style and Chameleon-style baselines under matched data, parameter, and optimization budgets. The project includes a paper, source code, and project page.

In-depth insight

Multimodal Flow offers a third route between fully discrete tokenization and hybrid discrete-continuous systems, using one continuous generation objective for multiple multimodal tasks.

机器之心 2026-11-05 16:21 北京

连续统一多模态模型的想象空间,或许才刚刚打开。

走向物理 AI 时代,构建一个能真正统一语言与视觉的多模态基座模型是关键一步。然而,语言本质上是有序的 Token 序列,视觉则是连续的高维像素分布,两者在信息结构上存在天然的鸿沟。如何在一个模型内统一这两种截然不同的分布,一直是一个备受争论的话题。

目前,主流的解法主要有两条。一条是全离散路线(如 Chameleon、Emu3),它把图像也量化成离散 Token,与语言拼在一起做自回归建模。但这种方法受限于视觉 Tokenizer,图像的高频细节在量化过程中极易丢失。另一条是混合离散 - 连续路线(如 Transfusion、JanusFlow),文本走离散预测,图像走连续扩散。虽然避开了量化瓶颈,但一个模型里得同时维护两套截然不同的目标函数和采样流程,不仅工程实现复杂,联合优化的难度也很大。

#Multimodal Flow#MF-1#Huazhong University of Science and Technology#Horizon Robotics#SigLIP2#T5-small#Chameleon#Emu3#Transfusion#JanusFlow

Original · 7 min readRead the original
Group chat

No comments yet. Start the conversation.