TokenSpeed: open-source inference engine for agentic workloads
TokenSpeed, an open-source LLM inference engine from the non-profit LightSeek Foundation, is trending on GitHub; it is pitched as a "speed-of-light" engine for agentic workloads aiming for TensorRT-LLM-level performance with vLLM-level usability. Its architecture is described as the first to separate a C++ control plane, implemented as a finite-state machine that uses the type system to enforce request lifecycle and KV cache resource safety at compile time, from a Python execution plane, with kernels handled as pluggable modules behind a portable API and centralized registry. The project's blog says TokenSpeed joined the PyTorch Ecosystem and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300 and Kimi K3 inference work on AMD Instinct MI355X, with engineering support from NVIDIA and AMD; those performance figures are self-reported.
A neutral, foundation-backed inference engine that splits control and execution planes could shape the cost and ecosystem choices for scaling agent deployments.