Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

ConvexLLM maps audio directly into an LLM's word-vector space

AI summary

CUHK-Shenzhen and Microsoft Research proposed ConvexLLM, a multimodal architecture that maps audio into a pretrained large language model's frozen word-vector convex hull without first transcribing speech into text. Its C-Gate generates audio representations through nonnegative combinations of frozen word vectors, enabling direct use of the LLM's pretrained knowledge and reasoning capabilities. AIbase reports strong VoiceBench results and an 8.8-point lead over the strongest baseline on BBH, but the supplied report does not independently verify these claims.

In-depth insight

ConvexLLM offers a distinct route to unified speech-language modeling by removing the transcription step and connecting audio directly to a pretrained LLM.

港中深联合微软研究院提出 ConvexLLM:让声音跳过转写,直接调用大模型推理能力

Image 1: aibase

AIbase基地

发布于AI新闻资讯 · 1 分钟阅读 · Nov 19, 2026

8

在语音与多模态人工智能的演进中,如何让大模型更自然、更直接地“听懂”声音并完成复杂推理,一直是业界探索的核心方向。近期,香港中文大学(深圳)与微软研究院联合提出了名为 ConvexLLM 的多模态统一建模新架构,为语音与语言的融合提供了全新的技术思路。

#ConvexLLM#C-Gate#The Chinese University of Hong Kong (Shenzhen)#Microsoft Research#VoiceBench#BBH

Original · 6 min readRead the original
Group chat

No comments yet. Start the conversation.