SageMaker AI推出KV缓存感知路由
AWS Machine Learning Blog称,Amazon SageMaker AI实时端点现已支持面向大模型服务的KV缓存感知路由。系统依据模型容器上报的缓存状态,把请求发送到仍保留匹配缓存且负载不高的实例。启用该功能需要设置SM_VLLM_KV_EVENTS_CONFIG环境变量、选择KV_AWARE路由策略,并使用受支持的容器。
与仅从请求推断缓存位置的前缀感知路由相比,此功能跟踪容器上报的缓存状态,并可识别内存压力下的缓存淘汰。
Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy to KV_AWARE in the endpoint configuration. Your clients call the endpoint with the same APIs as before.
Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In part 1 of this series, we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance. Prefix-aware routing infers where a prefix is cached from the request alone, so it can’t tell when an instance has evicted that cache.
#Amazon SageMaker AI#AWS vLLM Deep Learning Container for SageMaker AI#vLLM#Amazon Web Services