Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

vLLM expands encoder disaggregation for multimodal serving

AI summary

vLLM now supports Encoder / Prefill / Decode disaggregation, with encoder, prefill, and decode stages deployed independently or with prefill and decode combined. Recent work adds metadata-only requests for DeepSeek V4.1 Flash, remote embedding transfer through NIXL and Mooncake, shared caching, dedicated encoder execution, and orchestration integrations; core capabilities are included in vLLM v0.32.0. The vLLM Blog also reports approximately 94–115% higher end-to-end throughput in a 30-frame MMVU evaluation using Qwen3.5-35B-A3B on separate 4-GPU GB200 nodes, but the result comes from the project’s own testing.

In-depth insight

Separating multimodal encoding from prefill and decode lets operators scale each stage independently and transfer embeddings without repeating media preprocessing, giving vLLM users more flexible serving architectures.

Multimodal inference adds an encoding stage before language-model execution. Images, video frames, and audio must be processed into embeddings before the language model can use them. The demand on this stage varies with media size, modality, and request mix, and can change independently of prefill and decode demand.

Encoder / Prefill / Decode (EPD) disaggregation separates these stages into independently deployed vLLM instances. Recent work in the vLLM community has expanded EPD across model execution, remote embedding transfer, shared caching, multimodal input handling, and request routing.

DeepSeek's inference deployment Q&A describes an E/P/D architecture with a separately deployed vision encoder. vLLM has supported encoder disaggregation since our December 2025 architecture post, and continues to improve that support. vLLM main now also supports metadata-only EPD for DeepSeek V4.1 Flash.

This post explains the engineering behind that progress, with a focus on five changes:

  • Metadata-only requests that let downstream instances consume remote embeddings without repeating media preprocessing.
  • Dedicated encoder execution with a separate Model Runner V2 path, encoder-only CUDA Graph support, and reduced checkpoint-loading I/O.
  • Remote embedding transfer through NIXL and Mooncake integrations.
  • Cross-encoder output reuse through Mooncake Store.
  • Lower orchestration overhead in the Python EPD proxy and EPD orchestration with vLLM Router, SMG, and llm-d.

#vLLM#vLLM v0.32.0#DeepSeek V4.1 Flash#Qwen3.5-35B-A3B#NIXL#Mooncake#Mooncake Store#vLLM Router#SMG#llm-d#MiMo-V2-Omni

Original · 12 min readRead the original
Group chat

No comments yet. Start the conversation.