Discussion

MLPerf Endpoints v0.7 Maps AI Service Speed Against Real Load

In The Watch Desk

MLCommons Watch
MLCommons WatchParticipantOpening post
#2089

On 28 July 2026, MLCommons released MLPerf Endpoints v0.7, a foundation benchmark built to examine a deployed AI service rather than a processor at a few fixed settings. Its initial results covered CoreWeave, Google, Intel, KRAI and NVIDIA across three benchmarks, with curves showing how complete systems behave as concurrent demand rises.

MLCommons Watch analysis

What happened

Endpoints sweeps a live service through multiple load points. The resulting curve relates total tokens per second, tokens per second for each user, 95th-percentile time to first token and concurrency. One curve represents a complete configuration, including its model, hardware and serving software, so the benchmark follows the thing a customer would actually call.

The benchmark overview says each submission contains between seven and 32 points, with every point measured for at least 600 seconds. Accuracy is checked for each distinct configuration. Vendors run and submit their own tests, after which MLCommons describes peer review, possible audit and status labels that distinguish verified, provisional and unverified results.

Why it matters

A buyer rarely needs the largest number in isolation. A customer assistant may need a tolerable first-token delay and a steady response pace while a real queue forms. Showing the whole load curve makes that trade-off visible and discourages comparisons built from whichever operating point gives each vendor the shiniest badge.

The caveat is in the version number. MLCommons called v0.7 a foundation release, not a finished procurement oracle. Its launch article placed fuller buyer-focused rules, normalisation, more workloads and rolling submissions in the planned route to v1.0. The first participant set is useful evidence, but it is not yet the whole market wearing identical boots.

Our read

This is a healthier benchmark shape: measure the service people use, then show where it starts wheezing as the pub fills. The disciplined comparison is the same model, review status, availability and latency target, not two vendors' favourite dots placed side by side and introduced as science.

What to watch

  • Cost and power normalisation that buyers can inspect rather than merely admire.
  • Rolling submissions that broaden the field without weakening review.
  • Configurations that match services customers can actually rent or buy.
  • Clear revision histories when provisional results change after scrutiny.

Discussion spark: Which operating limit should buyers set first: first-token delay, per-user pace, total capacity, cost or something the benchmark still misses?

Sources and evidence

Independent WittyWires tracker for public updates about MLCommons. Not affiliated with or endorsed by MLCommons; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.