As agentic AI workloads grow, companies are trying to get more useful work from each unit of computing rather than simply buying more capacity. An Uber infrastructure leader said the company stabilised token costs while expanding agent use, using caching and a 24-million-node internal context graph to speed up queries.
Watch Desk analysis
What happened
The Information’s panel brought together speakers from Nebius, Uber and Nvidia to discuss practical ways to improve AI infrastructure efficiency. Uber’s Mattie Toia said its internal context graph, built from sources including Jira tickets, design documents and code reviews, cut query times from 20 minutes to 30 seconds while using fewer tokens.
The panel also described other approaches: Nebius uses speculative decoding in its inference platform, while Nvidia’s Dynamo can find previously computed data in a cluster’s KV cache instead of recalculating it. Nvidia’s Dion Harris said software and hardware improvements together can deliver three to five times the performance over a chip’s lifetime, from day one to year five. These are claims and examples attributed to the panel, not a like-for-like independent comparison.
Why it matters
More AI use does not have to mean a matching rise in compute bills. Caching can avoid repeated work, while better access to an organisation’s own information can reduce the effort an agent needs to find useful context. That makes infrastructure choices and data organisation part of the economics of deploying AI, not merely back-room housekeeping.
The panel’s shift from “token maxing” to “value maxing” is the useful idea here: count the cost of getting an outcome, not just the tokens consumed along the way. The 24-million-node graph is a striking example, though the reported time saving comes from Uber’s own account and may not transfer neatly to another organisation.
Our read
This is a refreshingly practical conversation about making AI work pay its way. Before adding capacity, teams should look at repeated computation, access to their own data and whether they can see where usage is going. Better plumbing is not glamorous, but neither is an invoice that keeps learning new tricks.
What to watch
- Whether Uber shares more detail on how it measures the context graph’s performance and token savings.
- How widely inference platforms adopt caching and speculative decoding, and what trade-offs they report.
- Whether managed services start routing workloads towards outcomes and cost, rather than simply processing more tokens.
Discussion spark: Should AI teams prioritise better data access and compute efficiency before buying more capacity, or does that risk slowing useful deployment?
Sources and evidence
- AI Agenda Live: From Token Maxing to Value maxing: Getting More From Every Unit of Compute (7 October 2026, 15:34 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.