SGLang HiCache brings KV reuse to distributed LLM serving
LMSYS Blog details how SGLang’s HiCache extends radix-cache prefix KV reuse beyond GPU memory to host memory and external storage. It supports context and pipeline parallelism, speculative decoding, unified KV layouts, and prefill-decode disaggregation, while SGLang’s rank_consensus_checker helps diagnose inconsistent cache decisions across ranks. The Iflytek Astron MaaS team reports that combining HiCache with external storage raised its cache hit rate from about 30% to 95% and reduced median TTFT to below three seconds.
HiCache addresses how cached-prefix reuse can remain consistent across distributed serving configurations, a core infrastructure concern for long-context inference workloads.
On this page
- 1. CP × HiCache: Page-based KV sharding
- 1.1 Logical radix tree and page ownership
- 1.2 Reading sharded KV: Gather one layer at a time
- 1.3 HiCache integration: Restore each shard before gathering
- 1.4 Performance practices
- 2. PP × HiCache: Keeping cache operations consistent across stages
- 2.1 Host cache: Coordinating asynchronous transfers
- 2.2 External storage: Agreeing on the cached prefix
- 3. Speculative decoding × HiCache
- 4. Unified KV cache layout: Reuse across compatible topologies
- 5. P/D × HiCache: Decode-side prefix reuse
- Closing perspective
- Debugging with rank_consensus_checker
- Acknowledgements
#SGLang#HiCache#Iflytek Astron MaaS#GLM-5.1#GLM-5.2 CodingPlan#rank_consensus_checker