Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

SageMaker AI adds KV cache-aware routing for real-time endpoints

AI summary

AWS Machine Learning Blog reports that Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new RoutingStrategy option for LLM serving. The feature uses cache state reported by the model container to send requests to instances holding matching cached context while avoiding overly busy instances. Adoption requires the SM_VLLM_KV_EVENTS_CONFIG environment variable, the KV_AWARE routing strategy, and a supported container.

In-depth insight

Unlike prefix-aware routing, which infers cache placement only from request content, the new strategy follows reported cache state and can account for cache eviction under memory pressure.

Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy to KV_AWARE in the endpoint configuration. Your clients call the endpoint with the same APIs as before.

Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In part 1 of this series, we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance. Prefix-aware routing infers where a prefix is cached from the request alone, so it can’t tell when an instance has evicted that cache.

#Amazon SageMaker AI#AWS vLLM Deep Learning Container for SageMaker AI#vLLM#Amazon Web Services

Original · 19 min readRead the original
Group chat

No comments yet. Start the conversation.