NVIDIA shared a process for finding issues in real agent traces, creating evaluations that reproduce failures, and revising instructions or tool execution. Agent developers can iterate by comparing task success rates and tool use before and after changes.
1. Find issues to fix in execution traces
Generic benchmarks alone rarely explain why an agent failed at a specific task. The harness, which combines instructions, tools, memory, and execution order, may also be responsible. NVIDIA's tutorial links trace analysis, evaluation, and harness changes in one cycle.
The example uses Hermes, an enterprise assistant at a fictional company. Asked to draft a project status message, the agent wrote it and then called a sending tool without user confirmation. Traces show what it did, and evaluations test whether a fix prevents recurrence.
The optimization blueprint uses three libraries: NeMo Compass, NeMo Eval Author, and NVIDIA NeMo Relay. Compass analyzes saved traces and evaluation evidence for recurring issues, coverage gaps, and priorities. The example uses 42 repository traces and ETHOS.md, which defines expected agent behavior.
compass --config configs/trace-analyst.yaml \
--trace.filesystem.path traces/world-v3/production/insights.jsonlCompass examines unusual behavior, tool use, and deviations from expectations, linking findings to trace identifiers. NVIDIA says applying all signals to thousands of internal benchmark traces found anomalies missed by a coding agent doing similar analysis.
2. Turn discovered failures into reproducible evaluations
NeMo Eval Author provides evaluation-authoring skills and workflows for coding agents. Skills are instructions for recurring tasks. It supports Harbor and NVIDIA NeMo Gym evaluations. Authors supply Compass findings, original traces, expected behavior documents, and the agent repository, then review whether evaluations measure the required behavior.
In the example, a coding agent created four Harbor evaluations resembling response-drafting tasks. The validation logic checks for calls to the sending tool without user confirmation and marks such runs as failures.
Even without real traces, evaluation can start from requirements, tool definitions, and existing workflows. The evaluation suite can expand as production traces and known failures accumulate.
3. Fix the harness and check the results
With evaluations in place, developers fix harness issues in instructions or tool execution code. The example changes Hermes's SOUL.md so drafting requests allow preparation only, with a review draft shown before stopping short of sending. It also clarifies that a preparation tool returning an execution token does not imply user consent.
Two development tasks and two held-out tasks hidden from the coding agent proposing changes were each run three times. NVIDIA reports that both evaluation suites had more passes and fewer tool calls after the instruction change.
Table: Hermes evaluation results before and after the instruction change
Conditions: Hermes enterprise assistant, 2 tasks per evaluation suite × 3 runs, held-out tasks hidden from the agent proposing changes
| Evaluation suite | Passes/runs before | Passes/runs after | Tool calls before (count) | Tool calls after (count) |
|---|---|---|---|---|
| Development | 3/6 | 6/6 | 10 | 6 |
| Held-out | 4/6 | 6/6 | 28 | 15 |
Testing hidden tasks helps reveal whether a fix is tailored only to development cases. NVIDIA recommends adopting changes when both suites improve, and repeating analysis with new traces if results are unclear or other behaviors worsen.
When Compass analyzed the revised traces, it no longer reported sending without confirmation. Instead, it found that the agent located answers but failed to support its responses with retrieved evidence, setting the next evaluation and improvement target.
4. Measure quality and efficiency in engineering agents too
NVIDIA also presented Dassault Systèmes' engineering assistant LEO proposing car door design changes and testing them in simulations. Compass found repeated placement of QueryModel arguments outside the designated options section. This produced empty error messages that the agent could not recover from.
After a coding agent added validation for misplaced arguments and similar issues, NVIDIA reported that invalid tool calls fell from 142 to 0. Accuracy remained unchanged, and no performance degradation was observed in the other scenarios.
The tutorial cites reported results: an initial run continued for about 12 hours without human intervention. Across six tested scenarios, the door design workflow's overall evaluation score improved by 28% relative, with 35% fewer tool calls per run.
This process suggests room to improve task quality and execution efficiency by revising instructions and tool validation without changing the model. It also adds engineering work to collect traces, expand evaluations, and monitor held-out cases and regressions in other behaviors.
Turning traces into reproducible evaluations can connect agent failures to harness issues that can be fixed.