vLLM and SGLang launch native PyTorch backends for Google Cloud TPUs
vLLM released vLLM TorchTPU, an out-of-tree platform plugin that lets vLLM run on Google Cloud TPUs, and Google said it also launched a native PyTorch backend for SGLang built with RadixArk. Both engines are built on Google's TorchTPU stack, reuse existing model definitions and keep upstream schedulers, KV cache managers and OpenAI-compatible servers, with TPU-specific Pallas kernels for attention, mixture-of-experts and quantized matmuls. Google says the engines are running in production with preview customers and cites SemiAnalysis InferenceX results including 2.01 cents per million tokens, while vLLM lists TPU v7x (Ironwood) as recommended hardware and v6e, v5e and v5p as experimental.
The release gives PyTorch developers a native route to serve popular open models on TPU without rewriting model code, while retaining existing vLLM and SGLang serving interfaces; performance and production claims are vendor-sourced.
Today we're releasing vLLM TorchTPU (vllm-torchtpu), a new backend for running vLLM on Google Cloud TPUs. It's built on Google's TorchTPU stack and runs the model definitions that already live in vLLM, so most models that work in vLLM today can be served on TPU without changes to the model code.
This work comes out of the engineering partnership between Inferact and Google we announced last month. Our goals for the project are to:
- Make it easy to bring new models to TPU by reusing vLLM's existing model code.
- Push TPU inference performance in open source, with Pallas kernels for attention, Mixture-of-Experts, and quantized matmul.
- Keep the vLLM experience the same: the same CLI, OpenAI-compatible server, scheduler, and KV cache manager you use on other hardware.
How It Works
vllm-torchtpu is an out-of-tree vLLM platform plugin. Once installed, vLLM picks it up automatically through the vllm.platform_plugins entry point.
#vLLM#SGLang#Inferact#RadixArk#Google Cloud TPU#PyTorch#TorchTPU#Meta#Qwen#DeepSeek#Kimi#GLM