创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

NVIDIA NeMo将智能体故障转化为测试与框架修复

AI 总结

NVIDIA技术博客教程演示了一套智能体优化流程:使用NeMo Compass从记录的追踪数据中找出问题,用NeMo Eval Author创建评测,更新智能体指令或框架,并在开发任务和保留任务上比较结果。NemoClaw示例在NVIDIA OpenShell沙箱中使用Hermes Agent及一家虚构公司的追踪记录。在另一个车门设计示例中,NVIDIA报告称,为错误工具调用增加验证后,此类调用从142次降至零。同一示例据称在六个测试场景中使工作流总体评测得分相对提升28%,每次运行的工具调用减少35%,其他场景未报告回归。

深度见解

该教程展示了如何利用生产或记录证据指导智能体评测和针对性框架改进,其范围不止于模型调优。示例还把评测与任务质量和工具使用效率等工程结果联系起来。

Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code.

When you build an agent for your domain, generic benchmark scores may not explain its failures. An agent might choose the wrong tool, follow the wrong workflow branch, miss relevant context, use an unnecessarily expensive model, or complete a task without meeting a domain-specific requirement.

Improving an agent requires more than model tuning. Prompts, tools, routing, memory, skills, control flow, and other code all shape production reliability. Traces help you investigate failures, while evaluations help you measure whether a change improves agent behavior.

This tutorial walks you through one complete optimization cycle: use NVIDIA NeMo Compass to identify a failure in recorded traces, create an evaluation with NVIDIA NeMo Eval Author, update the agent’s instructions, and compare results on development and held-out tasks. Follow along with the NVIDIA NemoClaw blueprint for agent optimization.

The blueprint uses Hermes Agent in an NVIDIA OpenShell sandbox, with recorded traces from a fictional company.

The agent harness is the optimization surface

One wrong tool call can derail an entire task. The cause may be an unclear tool description, a routing rule, or missing context in the agent harness. A trace helps identify where the run went wrong. Our example agent is a Hermes enterprise assistant for a fictional company, connected through Model Context Protocol (MCP) to the company’s chat, email, files, and tickets. We asked it to “draft a message with project updates.” It drafted the message but also called the send_message tool without user confirmation. Addressing this error requires examining both the harness instructions and tool execution.

#NVIDIA#NeMo Compass#NeMo Eval Author#NVIDIA NeMo Relay#NemoClaw#Hermes Agent#NVIDIA OpenShell#Model Context Protocol#Dassault Systèmes#LEO

原文约 7 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。