Evaluating an AI agent should start with what it needs to achieve, not a tempting list of metrics, AWS says. Its new guide shows how to turn business goals into observable behaviours, then measure them with Amazon Bedrock AgentCore Evaluations.
AWS AI Watch analysis
What happened
The guide recommends defining success in business terms, breaking it into observable behaviours and choosing evaluators at either trace or session level. For a support agent, that could mean checking whether it solved the customer’s problem, stayed coherent across turns and used the right tools.
AgentCore offers AWS-built evaluators, managed DeepEval and AutoEval options, and custom evaluators based on existing ones. AWS also describes multi-dimension evaluators, which score separate criteria in one judge call, and three ways to run evaluations: on demand, against sampled live traffic or in batch against a fixed dataset. Read AWS’s evaluation guide.
Why it matters
A single score can conceal a serious failure: an agent might resolve a support ticket while exposing personal information. AWS’s advice is to measure behaviours that matter independently, rather than trusting a flattering blended result.
That gives developers a practical starting point for testing agents before and after deployment. The guide also describes how teams already using DeepEval or AutoEval can run those metrics through AgentCore’s managed service.
Our read
The most useful advice is also the least glamorous: decide what success means before choosing how to score it. Otherwise, teams can end up with tidy numbers measuring the wrong thing, which is evaluation theatre with better dashboards.
Start with two or three behaviours that matter to your agent, then add evaluators when you find a failure your current set misses. AWS’s guide is a useful map, not evidence that any particular agent is reliable.
What to watch
- Whether teams choose measures tied to real outcomes, rather than convenient metric labels.
- How multi-dimension evaluations handle criteria that can fail independently.
- Whether teams validate evaluators against their own traffic and data before relying on their scores.
Discussion spark: When an AI agent is judged on several outcomes, should teams use one combined score for simplicity or keep each criterion separate, even if that makes release decisions harder?
Sources and evidence
- The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations (10 November 2026, 22:35 UTC)
not affiliated with or endorsed by Amazon Web Services (AWS)