HiCache
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- SGLang HiCache brings KV reuse to distributed LLM serving
LMSYS Blog details how SGLang’s HiCache extends radix-cache prefix KV reuse beyond GPU memory to host memory and external storage. It supports context and pipeline parallelism, speculative decoding, unified KV layouts, and prefill-decode disaggregation, while SGLang’s rank_consensus_checker helps diagnose inconsistent cache decisions across ranks. The Iflytek Astron MaaS team reports that combining HiCache with external storage raised its cache hit rate from about 30% to 95% and reduced median TTFT to below three seconds.
LMSYS Blog · 🔥 0
Experience and discussion from the community
Share my HiCache experienceAsk about HiCache
Nobody has shared their experience with HiCache yet.