vLLM与SGLang推出PyTorch TPU后端
vLLM 发布 vLLM TorchTPU，这是一个树外平台插件，使 vLLM 可在 Google Cloud TPU 上运行；Google 同时宣布与 RadixArk 合作推出 SGLang 的原生 PyTorch 后端。两个引擎都基于 Google 的 TorchTPU 栈，复用现有模型定义，并保留上游的调度器、KV 缓存管理器和 OpenAI 兼容服务器，同时用 Pallas 内核处理注意力、混合专家和量化矩阵乘等 TPU 专用计算。Google 称两个引擎已在预览客户的生产环境中运行，并引用 SemiAnalysis InferenceX 结果，包括每百万 token 2.01 美分；vLLM 则将 TPU v7x（Ironwood）列为推荐硬件，v6e、v5e 和 v5p 为实验性支持。
此次发布让 PyTorch 开发者无需改写模型代码即可在 TPU 上部署主流开放模型，同时沿用 vLLM 和 SGLang 原有的服务接口；相关性能与生产部署说法均来自厂商。
Today we're releasing vLLM TorchTPU (vllm-torchtpu), a new backend for running vLLM on Google Cloud TPUs. It's built on Google's TorchTPU stack and runs the model definitions that already live in vLLM, so most models that work in vLLM today can be served on TPU without changes to the model code.
This work comes out of the engineering partnership between Inferact and Google we announced last month. Our goals for the project are to:
- Make it easy to bring new models to TPU by reusing vLLM's existing model code.
- Push TPU inference performance in open source, with Pallas kernels for attention, Mixture-of-Experts, and quantized matmul.
- Keep the vLLM experience the same: the same CLI, OpenAI-compatible server, scheduler, and KV cache manager you use on other hardware.
How It Works
vllm-torchtpu is an out-of-tree vLLM platform plugin. Once installed, vLLM picks it up automatically through the vllm.platform_plugins entry point.
#vLLM#SGLang#Inferact#RadixArk#Google Cloud TPU#PyTorch#TorchTPU#Meta#Qwen#DeepSeek#Kimi#GLM