Switchyard: An open-source library that helps an AI agent choose which model handles each request.
What problem it solves
AI agents need to choose among models while balancing task success, cost, and latency for their workloads. Applications also need flexibility in where routing runs and who controls model clients, credentials, retries, and service operation.
Key features
Multiple routing algorithms — Routing can be configured with Auto, Task, Execution, or Composite strategies, including model-based classification and routing informed by recent tool activity.
Source: README
Embeddable routing core — Applications can embed libsy in a gateway or agent harness while retaining ownership of model clients, credentials, retries, and service operation.
Source: README
Streaming decision API — The Rust API exposes model calls, decision-model calls, and final routing outcomes through an asynchronous stream that applications can connect to their own clients.
Source: README, crates
Gateway and runtime integrations — Switchyard can be used through existing gateways or runners, including Agent Router, LiteLLM, NeMo Relay, and OpenRouter. Integration support varies by routing algorithm.
Source: README
Evaluation and tuning workflow — Users can compare routing configurations with their own model pool, including against a single-model baseline, and follow a documented tuning loop.
Source: README, benchmark
Quick start
Copied verbatim from the project's README.
let mut steps = algorithm.run_stream(request, models, state);
while let Some(step) = steps.next().await {
match step? {
Step::CallModel(call) => {
// Send call.request through your LLM client.
// Give the call future to call.respond(...).await.
}
Step::CallDecision(call) => {
// Send call.request through your decision-model client.
// Give its typed response future to call.respond(...).await.
}
Step::Done(outcome) => {
match outcome.response {
Some(response) => {
// Return this answer; no final LLM call is needed.
}
None => {
// Send outcome.request through your client for the final answer,
// or pass it to your gateway to make that call later.
}
}
}
}
}Who it may suit
This may suit developers building AI-agent gateways or harnesses that need configurable model routing while keeping control of model access and deployment.
Topics to discuss
Which routing strategy is most useful for your agent workloads and evaluation data?
How should teams measure completed-task cost rather than individual model-call cost?
Which gateway integration and deployment model best fits your application?
Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.
Read the original README https://github.com/NVIDIA-NeMo/Switchyard#readme
ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.
Comments
…