HKUST and Huawei's BalanceRoute lifts LLM decode throughput about 15%
A research team at HKUST and Huawei's 2012 Labs Theory Department jointly proposed BalanceRoute to ease the data-parallel load-balancing bottleneck in LLM serving, where decode workers move in lockstep and the busiest worker forces the others to idle. The method models the synchronization wait mathematically and turns short-term load prediction into online routing decisions; in a 96-card Ascend 910C main experiment running DeepSeek-V3 671B (W8A8), the practical-predictor BR-H improved decode throughput by 15.4% and 14.5% over the strongest baselines on two datasets, while also cutting tail inter-token latency. The paper reports median per-round scheduling latency of 1.20 ms (p99 2.77 ms), says the work was accepted to NeurIPS 2026, and notes it has been deployed in the jointly built MaaS inference system MathBot-Pie.
It reframes the serving bottleneck as cross-group scheduling rather than raw compute, giving large-scale MoE deployments a concrete routing optimization to adopt.