AWS’s applied AI chief says stop chasing AI ROI and measure the business instead

AWS’s applied AI chief Colleen Aubrey argues that businesses should judge AI by whether it changes the performance of the business, not by treating “AI ROI” as a standalone score. Her examples include a planning tool that AWS says helped one customer expand inventory oversight from 20 products to 20,000 while cutting inventory by 7 per cent.

AWS AI Watch analysis

What happened

In an interview with Business Leader published on 7 October, Aubrey said companies should keep measuring the business results they already care about, then look more closely at how AI might change their operations. Routine tasks such as drafting and summarising can be a useful starting point, she said, but she sees bigger potential in changing longstanding business bottlenecks.

One example is Amazon Connect Decisions, which Aubrey said gathers and prepares data, tests it against around 20 forecasting models, then presents a result for a human planner to examine. She said early launch partner Wells Vehicle Electronics went from paying attention to 20 SKUs to 20,000, and cut inventory levels by 7 per cent within the first few weeks. Those are figures Aubrey gave in the interview, not independently established results.

Aubrey also described Connect Customer’s Live Sync capability, used by United Airlines. In her example, a customer discussing a cancelled flight can see alternative bookings appear in the airline app while speaking with voice AI, then select an option on screen or aloud. She said she was not aware of AWS customers using these agents to make purchases or sign agreements.

Why it matters

The distinction is between using AI to speed up a task and using it to change what a business can manage. Aubrey’s inventory example makes that ambition concrete: reviewing far more products could change how planners spend their time and how much stock a company holds. Her account also puts a human planner in the decision loop, rather than presenting the forecast as an instruction to follow.

The examples come from an AWS executive discussing products her team builds, so they show the company’s approach and claimed customer results, not a neutral comparison of what AI tools achieve across businesses. Still, the operational details are more useful than another promise that AI will “transform” everything, a word that has lately been doing a great deal of unpaid overtime.

Our read

Aubrey’s advice is a good corrective to measuring AI for its own sake: start with the business problem and ask whether the result changes. But a metric only helps if it is specific enough to test. Businesses considering similar systems should ask what changed, over what period, and what evidence supports the result, while keeping a person able to inspect the reasoning and intervene.

What to watch

• Whether AWS publishes further detail on the measurement behind the Wells inventory figures.
• How Connect Decisions presents model choices and forecast reasoning to planners.
• Whether AI agents move from recommendations and assisted service towards purchases or agreements.

Discussion spark: Should businesses judge AI projects by improvements to existing business measures, or do they also need a separate way to measure what the AI itself contributes?

not affiliated with or endorsed by Amazon Web Services (AWS)