PyTorch Foundation

Headlines · 1

1. 20h
   TokenSpeed: open-source inference engine for agentic workloads

   TokenSpeed, an open-source LLM inference engine from the non-profit LightSeek Foundation, is trending on GitHub; it is pitched as a "speed-of-light" engine for agentic workloads aiming for TensorRT-LLM-level performance with vLLM-level usability. Its architecture is described as the first to separate a C++ control plane, implemented as a finite-state machine that uses the type system to enforce request lifecycle and KV cache resource safety at compile time, from a Python execution plane, with kernels handled as pluggable modules behind a portable API and centralized registry. The project's blog says TokenSpeed joined the PyTorch Ecosystem and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300 and Kimi K3 inference work on AMD Instinct MI355X, with engineering support from NVIDIA and AMD; those performance figures are self-reported.

   GitHub Trending（每日） · 🔥 6

Experience and discussion from the community

Share my PyTorch Foundation experienceAsk about PyTorch Foundation