vLLM首日支持Mistral Large 4
vLLM宣布首日支持Mistral Large 4，并通过专用的vllm/vllm-openai:mistral-large-4 Docker镜像提供部署。该实现支持FP8和NVFP4检查点、图像输入、内置DSpark推测解码、长上下文服务，以及使用ROCm的AMD Instinct MI350部署。相关工作还带来了上下文并行、基于NIXL的分离式服务和HiSparse主机内存KV卸载方面的通用改进。vLLM报告的性能与质量数据未在所提供材料中得到独立验证。
此次发布为Mistral Large 4提供了vLLM中的明确部署路径，涵盖多模态输入、多种检查点格式、推测解码、长上下文配置、分离式服务和AMD硬件支持。
We're excited to announce day-0 support for Mistral Large 4 in vLLM. Mistral Large 4 is Mistral's new frontier mixture-of-experts native multimodal model. It combines a DeepSeek-style sparse attention stack (MLA plus a DSA top-k indexer) with several new architecture features: cross-layer KV sharing, NoPE layers, SSMax and LeRoPE. It ships with a built-in DSpark speculative decoding drafter and a Pixtral vision encoder.
Supporting it took more than a model file. ML4 runs on vLLM's model runner V2 (MRV2) with breakable CUDA graphs. Its building blocks live in a new vllm/models/mistral/ package, and the work produced generic improvements to decode context parallelism (DCP), prefill context parallelism (PCP), DSpark, NIXL-based disaggregated serving and HiSparse host-memory KV offload.
Highlights
- Architecture: 72-layer MoE with 128 routed experts (top-4) plus 1 shared expert and a hidden size of 6144. Attention is MLA (KV LoRA rank 512, RoPE dim 64) with a DSA sparse indexer that selects the top 2048 keys per query. 1 trillion parameters in total, 49 billion active per token (52 billion including embeddings and output layers).
- Checkpoints: FP8 (block-scaled weights) and NVFP4 (NVFP4 experts, FP8 attention/shared experts). Both use an FP8 latent KV cache (
fp8_ds_mla) by default. - Multimodal: a Pixtral vision encoder for image inputs, including multiple images per prompt.
- Speculative decoding: a built-in DSpark drafter (3 sliding-window MLA layers, depth 3) accepts ~2.9 tokens per step and gives about 2x output throughput on TP8.
- Long context: DCP gives ~7.9x more KV capacity on TP8. PCP+DCP prefill cuts time to first token (TTFT) by up to 1.6x. DSpark works under both.
- Disaggregated serving: NIXL prefill/decode (P/D) disaggregation with DP/EP, TP and PCP×DCP prefill nodes, with DSpark on both sides.
- HiSparse: host-memory sparse KV offload, now with support for cross-layer KV sharing.
- AMD: day-0 on AMD Instinct MI350 Series GPUs with ROCm, including DSpark, image inputs and expert parallelism with MXFP4 experts.
#Mistral Large 4#vLLM#DSpark#Pixtral#NIXL#HiSparse#AMD Instinct MI350#Mistral AI#Red Hat