Private AI infrastructure should be sized around the work it must handle, not a guess at how many GPUs or employees are involved. A new guide uses a bank as its example to connect demand, memory, facilities and cost, giving operators a more grounded starting point than buying hardware first and asking questions later.
Watch Desk analysis
What happened
The guide, Private AI Sizing: Users, Agents, Rooms, Cost per Token, lays out a methodology for planning private AI infrastructure around workload characteristics and active requests. It uses Little’s law to estimate peak requests in flight, and says token-heavy agent loops need to be included in the planning.
It also covers memory for model weights and key-value caches, operational constraints, power and cooling limits, and the economics of cost per token. The framework is intended to help compare a private deployment with cloud alternatives.
Why it matters
A GPU count on its own says little about whether a system can handle real demand. Request patterns, the extra work generated by agent loops, memory requirements and the limits of the building all affect what a deployment can serve and what it costs.
That makes this useful beyond the infrastructure team: anyone weighing private AI against cloud services needs to know what workload they are paying to support. “Private” is not automatically cheaper, either. The guide’s focus on unit economics makes that comparison part of the sizing exercise rather than an afterthought.
Our read
The sensible move is to start with the work, then calculate what it asks of the system. That is less glamorous than picking a GPU fleet, but it is a better way to avoid expensive capacity that sits idle or a system that queues when people actually use it. The guide outlines the method; the supplied summary does not provide worked figures to apply to a particular organisation.
What to watch
- Whether teams can estimate peak demand from their own workload and request patterns.
- How agent loops change memory, capacity and cost assumptions.
- Whether the private deployment’s cost per token compares favourably with cloud alternatives.
- How power, cooling and operational limits shape the final design.
Discussion spark: When planning private AI, should teams begin with a detailed workload model, or is a simpler cloud-versus-hardware cost comparison good enough?
Sources and evidence
- Private AI Sizing: Users, Agents, Rooms, Cost per Token (29 October 2026, 12:55 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.