Amazon SageMaker AI Real-time Endpoints Add KV Cache-aware Routing
SageMaker AI real-time endpoints now support KV-aware routing, which routes each request to the instance holding the best cached match for its prompt, improving cache reuse and reducing latency for LLM applications with multi-turn conversations or long shared prompts.
What changed?
Amazon SageMaker AI real-time endpoints have introduced a new routing strategy called KV cache-aware routing. By enabling this, requests are routed to the instance that holds the longest cached match for the given prompt, leveraging cache state information reported by each model container. This strategy tracks hashed pointers to cached prompt segments in a managed cache index and is only active when an endpoint has at least two instances or inference component copies. KV-aware routing is currently supported with compatible containers (such as AWS vLLM Deep Learning Container) and only for requests using OpenAI-compatible Completions or Chat Completions formats.

Why does it matter to an everyday developer?
This update is important for developers building LLM applications, especially those that require multi-turn conversations or use long shared prompts, such as chat assistants or retrieval-augmented generation (RAG) systems. With KV-aware routing, cache reuse is more precise than in prefix-aware routing, since it takes actual cache state into account and adapts to cache evictions dynamically. This can significantly reduce latency and increase throughput under heavy load or with large prompts, saving both time and compute resources. Developers can more effectively utilize GPU memory for context reuse without manual concurrency or prefix length configurations.
What can the developer do now?
- 1
Update endpoint configuration
Set the RoutingStrategy parameter to KV_AWARE in your SageMaker endpoint configuration.
