AWS has published a guide to using decision models as custom evaluators in Amazon Bedrock AgentCore Evaluations. The practical appeal is a focused check that returns a probability over fixed answers, alongside broader assessments from large language models.
AWS AI Watch analysis
What happened
The guide shows how to connect AgentCore Evaluations to two decision-model backends: Jev, a hosted model from TypeSafe, and Strands Decider 2B, an open-source model. AWS describes three example checks: whether an agent response is grounded, how helpful it is on a four-level scale, and whether it completed a goal.
The integration uses code-based evaluators running in AWS Lambda to send agent traces to a model and translate its answer into a result AgentCore can use. AWS also outlines on-demand, batch and online evaluation modes. The AWS guide includes a sample HR assistant with mock data, rather than a reported customer deployment.
Why it matters
A fixed-choice evaluator can return a label and probability that teams can threshold or aggregate in code. That may suit narrow checks such as whether a response stayed on topic, without asking a general-purpose model to produce a free-form judgement every time.
AWS is clear that decision models complement, rather than replace, broader LLM-as-a-judge evaluators. The useful division is straightforward: use a narrow model for a defined question, and a more capable evaluator when the job needs reasoning across evidence or an explanation of what went wrong.
Our read
This is a practical integration pattern, not evidence that a new evaluator is automatically trustworthy. The guide’s strongest advice is also its least glamorous: test the measurement before you let it gate a release. AWS recommends comparing results against a labelled sample of your own traffic, with three independent human labelers and the existing evaluator applied to the same sessions.
That makes the feature interesting for teams already using AgentCore, particularly those who want to screen routine checks and reserve deeper evaluation for cases that merit it. The clever bit is not replacing judgement; it is making a narrow judgement easier to measure.
What to watch
- Whether teams publish validation results comparing decision-model scores with human labels.
- How Jev and Strands Decider perform on real agent traffic, beyond the guide’s mock-data example.
- Whether AWS adds further decision-model options or evaluator examples.
Discussion spark: Would you trust a calibrated decision-model score to gate an AI agent release after validation on your own traffic, or should a human review remain part of that decision?
Sources and evidence
- Use decision models with Amazon Bedrock AgentCore Evaluations (4 November 2026, 18:46 UTC)
not affiliated with or endorsed by Amazon Web Services (AWS)