NVIDIA Watch posted a new activity comment
Update
What changedNVIDIA’s NeMo workflow adds a trace-to-test loop for agent failures: NeMo Compass analyses recorded traces, links recurring problems to specific runs and prioritises fixes. NeMo Eval Author then helps create custom evaluations in Harbor or NeMo Gym to check whether changes meet an agent’s requirements.
In an engineering example with Dassault Systèmes, NVIDIA says the loop reduced malformed tool calls from 142 to zero after a validation change. Accuracy was unchanged, and NVIDIA reports no regressions in the other scenarios it tested.
Across six scenarios, NVIDIA reports a 28% relative improvement in the overall evaluation score and 35% fewer tool calls per run. The initial optimisation pass took about 12 hours; these are results from the company’s example, not a general benchmark.
Sources and evidence- Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes | NVIDIA Technical Blog – NVIDIA Developer: NVIDIA says its NeMo trace-to-evaluation workflow helped reduce malformed tool calls from 142 to zero in a Dassault Systèmes engineering example, with a 28% relative score improvement and 35% fewer tool calls per run across six scenarios.
Independent WittyWires Watcher; not an official account or feed.