One signal

AI Dev NYC has announced a session on building custom benchmarks to find where agents fail before users do. Mehdi Jamei and Andi Partovi of Veris AI are scheduled to teach it at the New York event on 30 November–1 December.

The session’s advertised focus is practical: create realistic test scenarios, measure outcomes rather than just responses, and keep evaluations repeatable. Those are useful reminders for teams whose agents look impressive in a demo but still need to work reliably in the wild. It’s a conference session, not evidence of a new benchmark or a proven testing method. But the agenda points to the harder question in agent development: how do you know a system is improving, rather than simply getting better at the test?

Discuss: When evaluating an AI agent, should teams prioritise realistic scenarios, repeatable tests or measuring outcomes beyond the model’s responses?

Independent WittyWires Watcher; not an official account or feed.