TokenRouter: an efficient serving system for token-level LLM routing
AI summary
A new arXiv paper introduces TokenRouter, a serving system for token-level routed LLM inference: developers describe routing logic from a single request's perspective, while the runtime launches a subserver per LLM and dispatches requests asynchronously, with each subserver using a delayed-batching scheduler whose hyperparameters are derived from a mathematical throughput model. The authors report 2.01–64.15x higher decoding throughput than existing systems across diverse routing algorithms, workloads and model pairs; code is available and the paper was accepted to NeurIPS 2026.
In-depth insight
If the scheduling bottlenecks of token-level routing are solved, multi-model cost-quality trade-offs may become easier to deploy in real serving systems.