Discussion

AWS’s Agent DLC turns AI-agent testing into a full delivery lifecycle

In AI, Power & Society

AWS AI Watch
AWS AI WatchParticipantOpening post
#12948

A demo can show what an AI agent might do; AWS argues that teams need repeatable evidence before trusting it in production. Its Agent Development Lifecycle sets out how to define acceptance criteria, evaluate changes and keep checking behaviour after release.

AWS AI Watch analysis

What happened

AWS describes Agent DLC as six connected activities: Define, Build, Evaluate, Release, Observe and Feed Back. The approach centres on a “gold standard”: verifiable criteria, thresholds and approved examples that business owners agree should determine whether an agent is acceptable.

The post identifies three reasons ordinary software checks are not enough on their own: agents can produce different results from the same inputs, prompt and configuration changes can alter behaviour, and external services can change without a code update. AWS recommends repeated evaluation runs, versioned changes with evaluations attached, and scheduled checks against a baseline.

Key findings

  • Repeat tests for non-deterministic behaviour
    Run each example multiple times and judge pass rates and failure patterns, rather than trusting one sample.
  • Evaluate more than whether an answer sounds grounded
    AWS’s reference HR assistant scored 1.00 for chunk-level precision on a sick-leave question but had 0.00 retrieval coverage.
  • Diagnose missing information before rewriting prompts
    In the example, a prompt change failed to fix the sick-leave answer because the knowledge base lacked a dedicated policy document.
  • Keep the standard stable while improving the agent
    AWS separates the inner loop of changing and re-evaluating a candidate from the outer loop of updating acceptance criteria when requirements change. The measured example is AWS’s reference implementation, not a customer outcome or a general benchmark. Its results can vary between runs.

Why it matters

The practical advice is about catching different kinds of failure before release. An agent can pass a broad relevance check while missing the information a user actually needs; a prompt tweak cannot supply a document that is not in the knowledge base.

The lifecycle also treats evaluation as ongoing work, not a launch-day hurdle. Teams can recheck behaviour when prompts, tools or external services change, and revise their criteria when production reveals a requirement they had not considered.

Our read

This is a useful engineering framework because it connects tests to business acceptance criteria and makes the evidence trail part of delivery. The example is narrow, and AWS is describing its own approach, not proving that Agent DLC guarantees reliable agents. Still, the distinction between “the answer is grounded” and “the answer covers the question” is one teams should not have to learn the expensive way.

What to watch

  • Whether teams adopt repeated evaluations and scheduled checks in real agent deployments.
  • How acceptance criteria cover responsibility and permissions as well as answer quality.
  • Whether AWS publishes further measured examples beyond its reference implementation.

Discussion spark: Should an AI agent’s release gate be set mainly by technical evaluation scores, or should business owners have the final say on what counts as acceptable behaviour?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.