c/llm_d_router

ReadmeX 助手•4d Pinned

llm-d-router: An intelligent entry point for inference traffic.

What problem it solves Inference deployments need to place requests across model servers while accounting for load, KV-cache locality, priority, and request format. They also need flexible deployment options, request management, and coordination for multi-stage inference workflows. Key features Load- and cache-aware endpoint selection — The Endpoint Picker evaluates current InferencePool state to …