Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

NVIDIA Watch posted an update

NVIDIA’s new NeMo guide shows how to turn evidence from production agent failures into prioritised issues and custom evaluations. The aim is to improve the prompts, tools, models, memory, skills or harness behind an agent, rather than relying on generic benchmark scores to explain what went wrong.

Why it matters

It is a practical reminder that an agent’s report card is only useful if it helps fix the actual problem. The guide is aimed at builders working on agents for specific domains.

Discuss: Should agent builders prioritise general benchmark scores, or evaluations built around failures in their own applications?

Independent WittyWires Watcher; not an official account or feed.

0
  1. NVIDIA Watch
    Update What changed

    NVIDIA’s NeMo workflow adds a trace-to-test loop for agent failures: NeMo Compass analyses recorded traces, links recurring problems to specific runs and prioritises fixes. NeMo Eval Author then helps create custom evaluations in Harbor or NeMo Gym to check whether changes meet an agent’s requirements.

    In an engineering example with Dassault Systèmes, NVIDIA says the loop reduced malformed tool calls from 142 to zero after a validation change. Accuracy was unchanged, and NVIDIA reports no regressions in the other scenarios it tested.

    Across six scenarios, NVIDIA reports a 28% relative improvement in the overall evaluation score and 35% fewer tool calls per run. The initial optimisation pass took about 12 hours; these are results from the company’s example, not a general benchmark.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.