AWS argues AI agents should be priced by the task, not the token

A cheaper model can cost more to run if it uses extra tokens, takes more turns or fails more often. AWS has published a practical guide to measuring the true cost of AI tasks and testing whether an agent is worth its keep.

AWS AI Watch analysis

What happened

The guide moves the comparison beyond token prices. AWS says the bill for a completed task can also depend on how efficiently a model tokenises the input, how much it writes, how many tool-use turns it needs and how much hidden reasoning it uses. Each extra turn can resend a growing conversation history, adding to the cost.

In AWS’s worked example, a model with twice the listed price ends up costing 65 per cent less for a task. The example assumes different token counts, output lengths and numbers of correction turns, so it is an illustration, not a promise about other workloads. AWS’s guide to cost per task and agent ROI also sets out ways to measure real usage and compare an agent with the human process it might replace.

Key findings

• Count the whole task
  Include every model call needed to produce a usable result, not just the first prompt and reply.
• Measure the hidden costs
  Tokenisation, verbose output, repeated tool calls and reasoning tokens can all change the bill.
• Benchmark on your own workload
  AWS recommends tracking token use and turns across actual tasks, then comparing candidates on the data they will handle.
• Set a human baseline first
  Record completion time, effort, quality and other relevant outcomes before the agent goes live.
• Count human review too
  AWS’s proposed measures include time spent reviewing work, as well as retries, escalations and whether the result was accepted.

Why it matters

A model’s price per token is easy to compare and can be a poor guide to the cost of getting useful work done. For teams buying or building agents, the guide offers a more grounded question: what does a successful task cost, including the retries and human attention around it?

The same discipline applies to claims about return on investment. If teams do not measure the existing workflow before deployment, they risk reconstructing a flattering baseline afterwards. AWS recommends using human baselines and controlled experiments; its guide is advice, not evidence that any particular agent saves money or improves outcomes.

Our read

This is the kind of unglamorous accounting that can stop an impressive demo becoming an expensive habit. Start by measuring a real task, including human review, and compare models on the same workload. A bargain token price is no bargain if the agent needs three attempts and a person to tidy up afterwards.

What to watch

• Whether teams publish task-level cost and success measures, rather than token prices alone.
• How often real workloads expose extra retries, lengthy outputs or human review.
• Whether measured gains hold up against a baseline recorded before the agent launches.

Discussion spark: When you assess an AI agent, should cost per successful task outweigh the model’s token price, even if that makes comparisons slower and more workload-specific?

not affiliated with or endorsed by Amazon Web Services (AWS)