Google Cloud TPU

Headlines · 2

1. Oct 20
   SGLang launches open-source PyTorch-native TPU serving backend

   SGLang launched SGLang-TorchTPU, an open-source native PyTorch backend that brings its serving stack to Google Cloud TPU while retaining the same core codebase used for GPUs. The project, developed by RadixArk’s SGLang team with Google and Meta, includes real-TPU CI tests and support for multiple language and diffusion models. SGLang and Meta personnel said the backend is running production inference at Meta, while Tom’s Hardware also reported the open-source release of a matching PyTorch-native vLLM version.

   LMSYS Blog · 🔥 0
2. Oct 20
   vLLM and SGLang launch native PyTorch backends for Google Cloud TPUs

   vLLM released vLLM TorchTPU, an out-of-tree platform plugin that lets vLLM run on Google Cloud TPUs, and Google said it also launched a native PyTorch backend for SGLang built with RadixArk. Both engines are built on Google's TorchTPU stack, reuse existing model definitions and keep upstream schedulers, KV cache managers and OpenAI-compatible servers, with TPU-specific Pallas kernels for attention, mixture-of-experts and quantized matmuls. Google says the engines are running in production with preview customers and cites SemiAnalysis InferenceX results including 2.01 cents per million tokens, while vLLM lists TPU v7x (Ironwood) as recommended hardware and v6e, v5e and v5p as experimental.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my Google Cloud TPU experienceAsk about Google Cloud TPU