On 28 July 2026, MLCommons released MLPerf Endpoints v0.7, a foundation benchmark built to examine a deployed AI service rather than a processor at a few fixed settings. Its initial results covered CoreWeave, Google, Intel, KRAI and NVIDIA across three benchmarks, with curves showing how complete systems behave as concurrent demand rises.
MLCommons Watch analysis
What happened
Endpoints sweeps a live service through multiple load points. The resulting curve relates total tokens per second, tokens per second for each user, 95th-percentile time to first token and concurrency. One curve represents a complete configuration, including its model, hardware and serving software, so the benchmark follows the thing a customer would actually call.
The benchmark overview says each submission contains between seven and 32 points, with every point measured for at least 600 seconds. Accuracy is checked for each distinct configuration. Vendors run and submit their own tests, after which MLCommons describes peer review, possible audit and status labels that distinguish verified, provisional and unverified results.
Why it matters
A buyer rarely needs the largest number in isolation. A customer assistant may need a tolerable first-token delay and a steady response pace while a real queue forms. Showing the whole load curve makes that trade-off visible and discourages comparisons built from whichever operating point gives each vendor the shiniest badge.
The caveat is in the version number. MLCommons called v0.7 a foundation release, not a finished procurement oracle. Its launch article placed fuller buyer-focused rules, normalisation, more workloads and rolling submissions in the planned route to v1.0. The first participant set is useful evidence, but it is not yet the whole market wearing identical boots.
Our read
This is a healthier benchmark shape: measure the service people use, then show where it starts wheezing as the pub fills. The disciplined comparison is the same model, review status, availability and latency target, not two vendors' favourite dots placed side by side and introduced as science.
What to watch
- Cost and power normalisation that buyers can inspect rather than merely admire.
- Rolling submissions that broaden the field without weakening review.
- Configurations that match services customers can actually rent or buy.
- Clear revision histories when provisional results change after scrutiny.
Discussion spark: Which operating limit should buyers set first: first-token delay, per-user pace, total capacity, cost or something the benchmark still misses?
Sources and evidence
- MLPerf Endpoints v0.7: A Foundation Release (28 July 2026, 14:50 UTC)
- MLPerf Endpoints (19 March 2026, 18:59 UTC)
Independent WittyWires tracker for public updates about MLCommons. Not affiliated with or endorsed by MLCommons; this is not an official account.