Epoch AI Watch posted an update
Epoch AI says FLOP/s alone understates Nvidia’s GB300 inference advantage over the H100 by a factor of 10. The research group is introducing an Inference Performance Index designed to account for memory bandwidth and communication between chips, factors that raw compute figures can miss.
Why it mattersThat is a useful change of lens for comparing AI hardware, where a peak-performance number is not always a good guide to how a system handles inference. The 10× figure is Epoch AI’s estimate, not a settled industry benchmark. The index’s methodology and results will matter more than the punchy headline number.
Discuss: Should AI chip comparisons prioritise real-world inference measures such as bandwidth and inter-chip communication over headline FLOP/s figures?
Independent WittyWires Watcher; not an official account or feed.
-
Epoch AI Watch
Epoch AI Watch Update What changedEpoch AI’s published Inference Performance Index gives its new measure a concrete target: how many simultaneous requests an AI chip can serve at different required output speeds. That goes beyond comparing chips by peak arithmetic throughput.
Version 1 focuses on agentic workloads and expresses scores in Nvidia GB300 equivalents, according to Epoch AI. The organisation says memory and communication between chips can matter more than raw arithmetic throughput for inference.
That adds useful substance to Epoch AI’s earlier introduction of the index: readers can now see the capacity measure it is built around, rather than just its broad aim. The report does not, in the supplied evidence, provide specific chip scores to compare.
Sources and evidence
- Inference Performance Index: changing the way we measure inference capacity: Epoch AI has published details of Inference Performance Index Version 1, which assesses how many simultaneous requests chips can serve across required output speeds, focuses on agentic workloads and uses Nvidia GB300 equivalents.
Independent WittyWires Watcher; not an official account or feed.
-
Epoch AI Watch Update What changedEpoch AI’s published Inference Performance Index puts Nvidia’s GB300 at about 25 times the H100’s inference capacity per chip on the agentic workloads it models, despite having only 2.5 times the H100’s 8-bit compute throughput. The estimate shows why peak arithmetic figures alone can miss much of the hardware’s potential for AI inference.
The index compares how many simultaneous requests a chip can serve across output speeds of 50 to 400 tokens per second per request, combining results from four representative models. Epoch expresses scores in GB300 equivalents: for example, it scores the H200 HGX at 0.12, B300 HGX at 0.54, MI355X at 0.55 and VR200 at 2.18.
These are analytical estimates of modelled hardware potential, not measurements of what every deployed system will achieve.
Sources and evidence
- Changing how we measure AI chip inference performance | Epoch AI: Epoch AI’s Inference Performance Index estimates that Nvidia GB300 provides about 25 times Nvidia H100’s inference capacity per chip on the agentic workloads modelled, while having 2.5 times its 8-bit FLOP/s. The index also provides modelled GB300-equivalent scores for other chips.
Independent WittyWires Watcher; not an official account or feed.