Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

vLLM adds day-0 support for Kimi K3.1 Pro

AI summary

vLLM announced day-0 support for Moonshot AI's Kimi K3.1 Pro across NVIDIA and AMD GPUs, Google TPUs, and Alibaba T-Head Zhenwu PPUs. The implementation adds sparse-attention support, a unified KV cache for MLA latent data, indexer keys and KDA state, and specialized decode and prefill paths. vLLM also reported long-context, throughput and reinforcement-learning benchmark gains, but these measurements have not been independently corroborated in the supplied report.

In-depth insight

The update gives deployment teams immediate access to Kimi K3.1 Pro on several accelerator platforms, while the new cache and sparse-attention paths target efficient long-context inference.

Kimi K3.1 Pro is Moonshot AI's most capable open-weight model: a 2.8T-parameter (104B active) multimodal MoE with a 1M-token context that matches the strongest closed-source models on coding, agentic and knowledge benchmarks.

The main change is attention: K3.1 Pro adds Sparse Attention (DSA) to all 24 full-attention layers for efficiency. vLLM supports it on day 0 on NVIDIA GPUs, AMD GPUs, Google TPUs and Alibaba T-Head Zhenwu PPUs.

#vLLM#Kimi K3.1 Pro#Moonshot AI#NVIDIA#AMD#Google TPU#Alibaba T-Head#Zhenwu PPU#FlashInfer#FlashMLA

Original · 17 min readRead the original
Group chat

No comments yet. Start the conversation.