Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

NVIDIA NeMo Turns Agent Failures into Tests and Harness Fixes

AI summary

An NVIDIA Technical Blog tutorial demonstrates an agent optimization cycle: use NeMo Compass to identify problems in recorded traces, create evaluations with NeMo Eval Author, update the agent’s instructions or harness, and compare results on development and held-out tasks. The NemoClaw example uses Hermes Agent in an NVIDIA OpenShell sandbox with recorded traces from a fictional company. In a separate car-door design example, NVIDIA reports that adding validation for malformed tool calls reduced such calls from 142 to zero. The same example reportedly achieved a 28% relative improvement in the workflow’s overall evaluation score and 35% fewer tool calls per run across six tested scenarios, with no regressions reported elsewhere.

In-depth insight

The tutorial shows how production or recorded evidence can guide targeted agent evaluation and harness changes beyond model tuning. Its examples also connect evaluations with engineering outcomes such as task quality and tool-use efficiency.

Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code.

When you build an agent for your domain, generic benchmark scores may not explain its failures. An agent might choose the wrong tool, follow the wrong workflow branch, miss relevant context, use an unnecessarily expensive model, or complete a task without meeting a domain-specific requirement.

Improving an agent requires more than model tuning. Prompts, tools, routing, memory, skills, control flow, and other code all shape production reliability. Traces help you investigate failures, while evaluations help you measure whether a change improves agent behavior.

This tutorial walks you through one complete optimization cycle: use NVIDIA NeMo Compass to identify a failure in recorded traces, create an evaluation with NVIDIA NeMo Eval Author, update the agent’s instructions, and compare results on development and held-out tasks. Follow along with the NVIDIA NemoClaw blueprint for agent optimization.

The blueprint uses Hermes Agent in an NVIDIA OpenShell sandbox, with recorded traces from a fictional company.

The agent harness is the optimization surface

One wrong tool call can derail an entire task. The cause may be an unclear tool description, a routing rule, or missing context in the agent harness. A trace helps identify where the run went wrong. Our example agent is a Hermes enterprise assistant for a fictional company, connected through Model Context Protocol (MCP) to the company’s chat, email, files, and tickets. We asked it to “draft a message with project updates.” It drafted the message but also called the send_message tool without user confirmation. Addressing this error requires examining both the harness instructions and tool execution.

#NVIDIA#NeMo Compass#NeMo Eval Author#NVIDIA NeMo Relay#NemoClaw#Hermes Agent#NVIDIA OpenShell#Model Context Protocol#Dassault Systèmes#LEO

Original · 7 min readRead the original
Group chat

No comments yet. Start the conversation.