SGLang HiCache扩展分布式KV复用
AI 总结
LMSYS Blog介绍，SGLang的HiCache将radix缓存的前缀KV复用从GPU内存扩展到主机内存和外部存储。它支持上下文并行、流水线并行、推测解码、统一KV布局及预填充与解码分离，rank_consensus_checker则用于排查各rank间的缓存决策分歧。科大讯飞Astron MaaS团队称，启用HiCache与外部存储后，缓存命中率从约30%升至95%，中位TTFT降至3秒以下。
深度见解
HiCache让分布式部署中的长上下文前缀复用能够覆盖多种并行与解码架构，是大模型推理基础设施的重要能力。
On this page
- 1. CP × HiCache: Page-based KV sharding
- 1.1 Logical radix tree and page ownership
- 1.2 Reading sharded KV: Gather one layer at a time
- 1.3 HiCache integration: Restore each shard before gathering
- 1.4 Performance practices
- 2. PP × HiCache: Keeping cache operations consistent across stages
- 2.1 Host cache: Coordinating asynchronous transfers
- 2.2 External storage: Agreeing on the cached prefix
- 3. Speculative decoding × HiCache
- 4. Unified KV cache layout: Reuse across compatible topologies
- 5. P/D × HiCache: Decode-side prefix reuse
- Closing perspective
- Debugging with rank_consensus_checker
- Acknowledgements
#SGLang#HiCache#Iflytek Astron MaaS#GLM-5.1#GLM-5.2 CodingPlan#rank_consensus_checker