AWS adds tests for whether AI agents choose the right skills and follow them

AWS has added skill-focused evaluation tools to Strands Evals and Amazon Bedrock AgentCore Evaluations. The important change is that teams can now inspect not only an agent’s final answer, but also whether it selected the correct procedure and followed that procedure properly.

AWS AI Watch analysis

What happened

In a post published by AWS on 22 September, the company describes three evaluators for agents that use modular skills. Skill Selection Accuracy checks whether the agent chose an appropriate skill for the task. Skill Instruction Following rates how completely it followed the selected skill’s prescribed steps. A deterministic Skill Invoked check confirms that a named skill was loaded at all.

AWS says the evaluations can work from a recorded Strands Evals trajectory or from OpenTelemetry traces collected through AgentCore Evaluations. The documented workflow includes an AgentCore CLI, and AWS says recognised skill-loading signals cover several agent frameworks, including Claude Code, the Claude Agent SDK, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands and Google ADK.

Why it matters

A fluent final answer can conceal a badly routed or incomplete job. AWS gives the example of an HR assistant that chooses a benefits skill for a dental-benefits question, or a PTO assistant that checks an employee’s balance but skips the rollover rules before submitting a request. The response may sound perfectly plausible while the process is wrong.

That distinction matters as agents move from answering questions to operating business procedures. Teams can now test whether an agent selected the right playbook, completed its required steps and loaded a critical skill. The checks also make it easier to identify the remedy: ambiguous skill descriptions may cause poor selection, while incomplete execution may require clearer instructions, a different skill design or a stronger model.

Our read

This is useful evaluation plumbing rather than a glamorous model launch, which is precisely why it deserves attention. AWS is pushing agent testing closer to the work people actually care about: did the task finish correctly, and did the software follow the required route along the way?

The evidence is AWS’s own guidance, not an independent assessment of accuracy or production performance. Teams should run the evaluators against their own traces and regression cases, especially for workflows involving compliance, finance, HR or sensitive documents. A benchmark that rewards polished nonsense is still nonsense, just wearing a nicer lanyard.

What to watch

• Coverage:
  whether more frameworks and skill formats are supported reliably.
• Trace quality:
  whether recorded trajectories contain enough evidence to judge each step.
• Deployment gates:
  whether teams use critical-skill checks to block unsafe regressions.
• False confidence:
  whether judge-based scores are paired with deterministic checks and human review. AWS’s specific relevance to WittyWires is concrete: the release adds evaluators for AI agents that select and execute modular skills, with results drawn from recorded agent trajectories and production traces.

Discussion spark: Should agent evaluations prioritise the final result, or should skipping a required procedure be an automatic failure even when the answer looks right?

not affiliated with or endorsed by Amazon Web Services (AWS)