vLLM adds day-0 support for Mistral Large 4
vLLM announced day-0 support for Mistral Large 4, Mistral’s native multimodal mixture-of-experts model, through the dedicated vllm/vllm-openai:mistral-large-4 Docker image. The implementation supports FP8 and NVFP4 checkpoints, image inputs, built-in DSpark speculative decoding, long-context serving and AMD Instinct MI350 deployment with ROCm. It also includes generic improvements to context parallelism, NIXL-based disaggregated serving and HiSparse host-memory KV offload. Performance and quality figures reported by vLLM were not independently verified in the supplied material.
The release provides a documented deployment path for Mistral Large 4 in vLLM, covering multimodal inputs, multiple checkpoint formats, speculative decoding, long-context configurations, disaggregated serving and AMD hardware support.
We're excited to announce day-0 support for Mistral Large 4 in vLLM. Mistral Large 4 is Mistral's new frontier mixture-of-experts native multimodal model. It combines a DeepSeek-style sparse attention stack (MLA plus a DSA top-k indexer) with several new architecture features: cross-layer KV sharing, NoPE layers, SSMax and LeRoPE. It ships with a built-in DSpark speculative decoding drafter and a Pixtral vision encoder.
Supporting it took more than a model file. ML4 runs on vLLM's model runner V2 (MRV2) with breakable CUDA graphs. Its building blocks live in a new vllm/models/mistral/ package, and the work produced generic improvements to decode context parallelism (DCP), prefill context parallelism (PCP), DSpark, NIXL-based disaggregated serving and HiSparse host-memory KV offload.
Highlights
- Architecture: 72-layer MoE with 128 routed experts (top-4) plus 1 shared expert and a hidden size of 6144. Attention is MLA (KV LoRA rank 512, RoPE dim 64) with a DSA sparse indexer that selects the top 2048 keys per query. 1 trillion parameters in total, 49 billion active per token (52 billion including embeddings and output layers).
- Checkpoints: FP8 (block-scaled weights) and NVFP4 (NVFP4 experts, FP8 attention/shared experts). Both use an FP8 latent KV cache (
fp8_ds_mla) by default. - Multimodal: a Pixtral vision encoder for image inputs, including multiple images per prompt.
- Speculative decoding: a built-in DSpark drafter (3 sliding-window MLA layers, depth 3) accepts ~2.9 tokens per step and gives about 2x output throughput on TP8.
- Long context: DCP gives ~7.9x more KV capacity on TP8. PCP+DCP prefill cuts time to first token (TTFT) by up to 1.6x. DSpark works under both.
- Disaggregated serving: NIXL prefill/decode (P/D) disaggregation with DP/EP, TP and PCP×DCP prefill nodes, with DSpark on both sides.
- HiSparse: host-memory sparse KV offload, now with support for cross-layer KV sharing.
- AMD: day-0 on AMD Instinct MI350 Series GPUs with ROCm, including DSpark, image inputs and expert parallelism with MXFP4 experts.
#Mistral Large 4#vLLM#DSpark#Pixtral#NIXL#HiSparse#AMD Instinct MI350#Mistral AI#Red Hat