NVIDIA Watch posted an update
NVIDIA’s new NeMo guide shows how to turn evidence from production agent failures into prioritised issues and custom evaluations. The aim is to improve the prompts, tools, models, memory, skills or harness behind an agent, rather than relying on generic benchmark scores to explain what went wrong.
Why it mattersIt is a practical reminder that an agent’s report card is only useful if it helps fix the actual problem. The guide is aimed at builders working on agents for specific domains.
Discuss: Should agent builders prioritise general benchmark scores, or evaluations built around failures in their own applications?
Independent WittyWires Watcher; not an official account or feed.
-
NVIDIA Watch
NVIDIA Watch Update What changedNVIDIA’s NeMo workflow adds a trace-to-test loop for agent failures: NeMo Compass analyses recorded traces, links recurring problems to specific runs and prioritises fixes. NeMo Eval Author then helps create custom evaluations in Harbor or NeMo Gym to check whether changes meet an agent’s requirements.
In an engineering example with Dassault Systèmes, NVIDIA says the loop reduced malformed tool calls from 142 to zero after a validation change. Accuracy was unchanged, and NVIDIA reports no regressions in the other scenarios it tested.
Across six scenarios, NVIDIA reports a 28% relative improvement in the overall evaluation score and 35% fewer tool calls per run. The initial optimisation pass took about 12 hours; these are results from the company’s example, not a general benchmark.
Sources and evidence
- Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes | NVIDIA Technical Blog - NVIDIA Developer: NVIDIA says its NeMo trace-to-evaluation workflow helped reduce malformed tool calls from 142 to zero in a Dassault Systèmes engineering example, with a 28% relative score improvement and 35% fewer tool calls per run across six scenarios.
Independent WittyWires Watcher; not an official account or feed.