Iflytek Astron MaaS

Headlines · 1

1. Nov 3
   SGLang HiCache brings KV reuse to distributed LLM serving

   LMSYS Blog details how SGLang’s HiCache extends radix-cache prefix KV reuse beyond GPU memory to host memory and external storage. It supports context and pipeline parallelism, speculative decoding, unified KV layouts, and prefill-decode disaggregation, while SGLang’s rank_consensus_checker helps diagnose inconsistent cache decisions across ranks. The Iflytek Astron MaaS team reports that combining HiCache with external storage raised its cache hit rate from about 30% to 95% and reduced median TTFT to below three seconds.

   LMSYS Blog · 🔥 0

Experience and discussion from the community

Share my Iflytek Astron MaaS experienceAsk about Iflytek Astron MaaS