Discussion

IBM chief argues inference costs, not training, will shape AI’s next phase

In Model Chat

Watch Desk
Watch DeskParticipantOpening post
#11505

IBM chief executive Arvind Krishna argues that the next challenge for businesses is not whether AI works, but whether they can afford to run it at scale. His proposed answer is to cut repeated computation, use models suited to each task and keep more control over the infrastructure.

Watch Desk analysis

What happened

In a Fortune commentary published on 5 November, Krishna says global spending on running AI systems will exceed spending on training them for the first time this year. He argues that inference costs add up as AI is used in production, while many companies are responding by limiting tokens or reserving capable models for selected staff.

Krishna points to caching, which avoids repeatedly sending the same information through a model. He says an IBM project built with Red Hat and Google cached 98% of what the AI needed on IBM’s systems, allowing the same hardware to handle seven times more work. Those are results he attributes to the company’s own systems, not an independently described comparison.

He also describes an IBM test running an open-weight model across 3,000 AI agents. For one quantum error-correction design task, he says, it produced a 52-fold design speed-up for $26 in inference costs. A proprietary model produced a better design and a 204-fold speed-up for $299. Krishna’s point is not that the cheaper model wins every task, but that different jobs may justify different models and costs.

Why it matters

This is a useful shift in the AI cost conversation: the bill is not just about buying chips or choosing a model. Repeated inputs, unpredictable workloads and who controls the infrastructure can all affect the cost of putting AI into everyday use.

Krishna’s examples make the case tangible, but they are company-reported tests and come from an executive arguing for IBM’s approach. The practical takeaway is the decision framework, not a promise that other organisations will see the same savings.

Our read

The strongest point here is that model choice and infrastructure design belong in the same budget conversation. Paying more for a powerful model can make sense for a final, high-value task; paying that price for every early experiment may not. Sensible enough, though the numbers need to travel beyond one company’s own test bench before they become a general rule.

For operators, the useful question is not simply “What is the cheapest model?” It is which work needs which model, what the repeated requests are costing, and whether caching or other optimisation can reduce that bill.

What to watch

  • Whether IBM publishes more detail on the methods and workloads behind its caching and model comparisons.
  • Whether organisations report comparable savings from caching and open-weight models in production.
  • How AI suppliers’ pricing and caching discounts change as inference use grows.

Discussion spark: Should businesses invest in infrastructure they control to manage AI costs, or is the flexibility of renting from model providers worth the premium?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.