AWS turns an agent’s production traces into guarded prompt improvements

AWS has detailed a Bedrock AgentCore workflow that uses production traces to suggest better system prompts, then tests those changes before they reach live users. The practical significance is a move from manually tuning agent behaviour towards a repeatable optimisation loop, with guardrails intended to stop the system making prompts longer, copying trace text or weakening safety rules.

AWS AI Watch analysis

What happened

The workflow starts with traces recorded through AgentCore Observability and a reward signal from evaluations. A reflector agent examines successful and failed runs, identifies patterns and proposes changes to an agent’s configuration. Developers can review the explanation, run offline evaluations and use controlled A/B tests before promoting a recommendation.

AWS describes two designs. The Single Agent Reflector studies the full trace set in one pass, while the experimental Sub-Agent Reflector divides the work across agents that inspect individual traces before an orchestrator combines the findings. The second approach is intended to catch less common failure modes that a single pass might miss.

The company reports results on the AppWorld and WebShop benchmarks. Its Single Agent Reflector reached 81.55% on AppWorld in six minutes and 20 turns, while the Sub-Agent Reflector reached 95.83% in 193 minutes and 100 turns. On WebShop, the two approaches scored 78.31% and 79.15% respectively. Those are AWS’s benchmark results, not a guarantee that the same gains will appear in a production system with messier tasks and less cooperative traces.

Why it matters

Agent builders usually discover poor behaviour the slow way: inspect a long trace, guess which instruction failed, edit the prompt and run the whole test again. AWS’s approach makes the trace corpus part of the optimisation process, allowing the reflector to search files, compare successful and unsuccessful runs and propose targeted changes.

The useful detail is that the system is not supposed to publish its own fixes directly. AWS says candidate changes face a 20% length ceiling, safety screening and a ban on copying exact phrases from the traces used for optimisation. Developers are still expected to review recommendations and validate them offline and through controlled live tests. The robot may suggest the edit, but it has not been granted the office keys.

Our read

This is one of the more grounded agent developments because it addresses the unglamorous question of how systems improve after deployment. The reported benchmark results are striking, particularly for the Sub-Agent Reflector on AppWorld, but they come from AWS’s own evaluation setup and should be treated as evidence of a promising method rather than a universal performance claim.

The strongest idea is the review loop: use real behaviour to find patterns, let an agent propose changes, then keep humans and tests between recommendation and production. That makes optimisation more auditable than letting a model quietly rewrite its instructions. It also creates a new failure surface, because bad traces, weak reward signals or narrow test sets could teach the optimiser precisely the wrong lesson. Every self-improving agent needs a sceptical editor, preferably one with access to the undo button.

What to watch

• Production evidence
  Whether AWS or customers publish results from live workloads rather than public benchmarks alone.
• Reward quality
  How the workflow behaves when the score is incomplete, noisy or easy to game.
• Guardrail failures
  Whether candidate prompts are rejected for safety or overfitting problems in real deployments.
• Open-source reflector
  What the experimental Sub-Agent Reflector makes available through the Strands repository, and whether others can reproduce the results.

Discussion spark: Should agent prompt optimisation be allowed to learn from live production traces, or should organisations require a separate, human-curated dataset before any suggested change can reach users?

not affiliated with or endorsed by Amazon Web Services (AWS)