AWS vLLM Deep Learning Container for SageMaker AI
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- SageMaker AI adds KV cache-aware routing for real-time endpoints
AWS Machine Learning Blog reports that Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new RoutingStrategy option for LLM serving. The feature uses cache state reported by the model container to send requests to instances holding matching cached context while avoiding overly busy instances. Adoption requires the SM_VLLM_KV_EVENTS_CONFIG environment variable, the KV_AWARE routing strategy, and a supported container.
AWS Machine Learning Blog · 🔥 0
Experience and discussion from the community
Share my AWS vLLM Deep Learning Container for SageMaker AI experienceAsk about AWS vLLM Deep Learning Container for SageMaker AI
Nobody has shared their experience with AWS vLLM Deep Learning Container for SageMaker AI yet.