TokenSpeed

Headlines · 2

1. 20h
   TokenSpeed: open-source inference engine for agentic workloads

   TokenSpeed, an open-source LLM inference engine from the non-profit LightSeek Foundation, is trending on GitHub; it is pitched as a "speed-of-light" engine for agentic workloads aiming for TensorRT-LLM-level performance with vLLM-level usability. Its architecture is described as the first to separate a C++ control plane, implemented as a finite-state machine that uses the type system to enforce request lifecycle and KV cache resource safety at compile time, from a Python execution plane, with kernels handled as pluggable modules behind a portable API and centralized registry. The project's blog says TokenSpeed joined the PyTorch Ecosystem and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300 and Kimi K3 inference work on AMD Instinct MI355X, with engineering support from NVIDIA and AMD; those performance figures are self-reported.

   GitHub Trending（每日） · 🔥 6
2. 3d
   Qwen3.8-Flash-Next hits 685 output tokens/s per user with TokenSpeed on GB300

   The LightSeek Foundation reports that Alibaba's Qwen3.8-Flash-Next reached 685 output tokens per second per user, running its TokenSpeed serving stack on 4× NVIDIA GB300 GPUs with NVFP4 precision. Alibaba's Qwen account amplified the result on X, thanking LightSeek and saying it is excited to see the community unlock more of Qwen's inference potential. LightSeek said a blog post details the kernel and serving optimizations behind the numbers, with the code available on GitHub.

   Qwen (X) · 🔥 0

Experience and discussion from the community

Share my TokenSpeed experienceAsk about TokenSpeed