创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

用NeMo Switchyard路由Bedrock模型上的编程智能体请求

AI 总结

一份指南介绍了如何在AWS上部署带有NVIDIA NeMo Switchyard路由插件的LiteLLM,让编程智能体请求在Amazon Bedrock上的不同模型之间路由。Stage路由会读取近期工具活动及请求历史结果,从两个候选模型中选择一个;Random路由则支持加权流量分配和A/B比较。示例基准测试运行器会将路由模型组的每个已解决任务成本与单模型基线进行比较。Switchyard LiteLLM集成仍属实验性质,示例代码仅供非生产环境使用,应部署在非生产账户中。

深度见解

报告指出,编程任务的成本取决于模型轮数、重放的历史、重试次数和未完成的尝试,因此仅看令牌价格并不可靠。在开发者自己的工作负载上衡量每个已解决任务的成本,可以为比较路由配置与单模型基线提供依据。

Coding agents such as Claude Code and OpenAI Codex CLI work in tasks. A developer asks for a change, and the agent reads files, edits code, and runs tests until the work is done. One task can take a few model calls or dozens. What the developer pays for is the task, not one call.

The price per token alone doesn’t tell you what a whole task costs. The total depends on how many turns a model needs, how much history each turn replays, and how often a task has to be retried or never finishes. You pay for every attempt, including attempts on tasks that never finish. So when you route a coding agent’s requests between models, judge the routing by cost per solved task: the total cost of your runs divided by the number of runs whose tests pass.

NVIDIA NeMo Switchyard is an open source library that helps an AI agent choose which model handles each request. You define the models that it can use and the strategy that it follows. Its LiteLLM integration supports two strategies. Stage routing reads the recent tool activity and results in the request history, and then picks one of two candidate models. Random routing picks from the configured models, with optional weights, for traffic splits and A/B comparisons. Another option you can explore is LiteLLM, who also recently shipped a native built-in Switchyard stage router as an auto-router strategy.

#NVIDIA#NVIDIA NeMo Switchyard#LiteLLM#Amazon Bedrock#AWS#AWS Cloud Development Kit (AWS CDK)#Amazon ECS#Amazon Fargate#Claude Code#OpenAI Codex CLI

原文约 18 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。