Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

vLLM v0.34.0 adds 10 models and enables extensible KV cache by default

AI summary

vLLM v0.34.0 introduces 10 models, including Mistral Large 4, MiniMax-M3.1-Flash and NVIDIA Nemotron-3.5-Transcribe, and makes extensible KV caching the default. The release removes Model Runner V1, requires huggingface_hub 2.0.0 or later, changes the default block size to 64 and drops ROCm 7.2 artifacts. vLLM reports up to 1.95× faster decoding, 1.6× higher throughput and 1.24–1.40× faster long-prompt TTFT on Blackwell and MI355X hardware.

In-depth insight

This is a significant inference-engine update that expands model support and changes important runtime defaults, but its reported Blackwell and MI355X performance gains remain vendor claims.

v0.34.0

1,061 commits from 412 contributors (120 new). Thank you!

TL;DR

  • 10 new models (Mistral Large 4, MiniMax-M3.1-Flash, Nemotron-3.5-Transcribe, and more!)
  • Extensible KV cache is on by default: vLLM sizes the KV cache from memory actually free after warmup, so most models get a bigger cache and more throughput - in most cases you should now remove gpu_memory_utilization from your configs.
  • Model Runner V1 is gone. Model Runner V2 is now the only GPU runner.
  • Big speedups on Blackwell and MI355X: up to 1.95× decode (Step5 on GB200), 1.6× throughput (GLM-5.3-Flash), 1.24–1.40× faster long-prompt TTFT (GLM-5.3).
  • Upgrade check: default block size is now 64, ROCm 7.2 images are gone, and huggingface_hub >= 2.0.0 is required. See Before you upgrade.

Before you upgrade

Important

Some source content is omitted. Continue reading on the original site.

#vLLM#Mistral Large 4#MiniMax-M3.1-Flash#NVIDIA Nemotron-3.5-Transcribe#Ricursive Ripple-65B Preview#Zyphra ZAYA1#Baidu Unlimited-OCR-2#Liquid AI LFM3#NVIDIA Canary-1B-v2#Ming3-ASR#NVIDIA Parakeet TDT

Original · 20 min readRead the original
Group chat

No comments yet. Start the conversation.