What problem it solves

Inference deployments need to place requests across model servers while accounting for load, KV-cache locality, priority, and request format. They also need flexible deployment options, request management, and coordination for multi-stage inference workflows.

Key features

• Load- and cache-aware endpoint selection — The Endpoint Picker evaluates current InferencePool state to choose request placement using signals such as KV-cache locality, load, and priority.

  Source: README

• Flexible deployment modes — The router can run with a self-managed Envoy proxy in sidecar or service topology, or operate in Gateway Mode behind an InferencePool and HTTPRoute.

  Source: README

• Request management APIs — InferenceObjective configures scheduling goals such as priority and performance targets, while InferenceModelRewrite supports model-name rewriting for traffic management.

  Source: README

• Disaggregation coordination — A sidecar coordinates specialized workers for P/D and E/P/D inference lifecycles, including KV-cache and embedding transfers.

  Source: README, docs/disaggregation.md

Who it may suit

This project may suit teams operating LLM inference services on Kubernetes that need request-aware endpoint routing, cache-aware placement, or multi-stage inference coordination.

Topics to discuss

• Which routing signals should be considered first when KV-cache locality, endpoint load, and request priority conflict?

• How should session affinity and endpoint pinning behave when pods become unavailable or their state changes?

• Which operational trade-offs should guide the choice between Standalone Mode and Gateway Mode?

Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/llm-d/llm-d-router#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.