创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

SageMaker AI推出KV缓存感知路由

AI 总结

AWS Machine Learning Blog称,Amazon SageMaker AI实时端点现已支持面向大模型服务的KV缓存感知路由。系统依据模型容器上报的缓存状态,把请求发送到仍保留匹配缓存且负载不高的实例。启用该功能需要设置SM_VLLM_KV_EVENTS_CONFIG环境变量、选择KV_AWARE路由策略,并使用受支持的容器。

深度见解

与仅从请求推断缓存位置的前缀感知路由相比,此功能跟踪容器上报的缓存状态,并可识别内存压力下的缓存淘汰。

Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy to KV_AWARE in the endpoint configuration. Your clients call the endpoint with the same APIs as before.

Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In part 1 of this series, we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance. Prefix-aware routing infers where a prefix is cached from the request alone, so it can’t tell when an instance has evicted that cache.

#Amazon SageMaker AI#AWS vLLM Deep Learning Container for SageMaker AI#vLLM#Amazon Web Services

原文约 19 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。