AI agents should be tested on the everyday work people actually ask them to do, not just tidy demo tasks. Golda Manuel’s new guide sets out five ways to build evaluations that reveal where an agent succeeds, struggles or needs better product support.
Watch Desk analysis
What happened
Manuel’s guide, published on Hugging Face on 5 November, explains how to turn real developer workflows into evaluation tasks. It covers choosing tasks, defining success, testing variations, checking the evaluation itself and keeping the dataset current. Read the guide on Hugging Face.
Why it matters
A single success score can hide whether an agent handles the situations users actually face. These steps make the evaluation more informative: teams can see which workflows fail, whether the scoring is fair, and whether a product change improves performance beyond the examples used to make it.
Our read
This is refreshingly practical advice: test the work, the awkward variations and the test itself. The useful shift is from asking whether an agent can complete a polished task to checking whether it can handle a real workflow reliably. Evaluation is less glamorous than a leaderboard, but it is where many product promises meet a slightly expired credential.
What to watch
- Whether teams publish task definitions and checks, not just headline scores.
- How often evaluation datasets are refreshed as products and workflows change.
- Whether held-back tasks catch regressions outside the cases used to tune an agent.
Discussion spark: Should AI teams publish their evaluation tasks and scoring checks, or would that make it too easy to optimise for the test?
Sources and evidence
- Choosing the Right Tasks for AI Agent Evaluation (5 November 2026, 21:36 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.