MLCommons adds modern recommendation models to its AI training benchmark

MLCommons has introduced DLRMv4, a new MLPerf Training benchmark for AI recommendation systems built around users’ sequences of past actions. It replaces an older benchmark design that flattened behaviour into aggregated features, giving hardware comparisons a workload closer to how large recommendation models are trained today.

MLCommons Watch analysis

What happened

DLRMv4 uses HSTU, a model architecture that processes a user’s interaction history as a sequence, and Yambda-5B, a real-world dataset released by Yandex Music. MLCommons says the dataset contains about 4.79 billion interactions across one million users and 9.39 million items.

The benchmark includes five kinds of behaviour: listening, liking, disliking, unliking and undisliking. Its embedding tables reach about 560 GB, and the maximum sequence length is 4,096. MLCommons describes the benchmark as filling a gap in MLPerf Training: the suite already covered an HSTU-based workload for inference, but not training.

Key findings

• Histories stay in sequence
  The model processes past items and actions chronologically instead of relying on a small set of aggregated behaviour features.
• The data reflects several kinds of feedback
  Yambda-5B includes actions such as listens, likes and skips, preserving distinctions that a single summary score could blur.
• The systems challenge is substantial
  Long histories and large embedding tables make the workload demanding to train and distribute across accelerators.

Why it matters

Recommendation models quietly shape what people see in shopping, music and streaming services. A benchmark that better reflects their real workloads can make comparisons between training systems more relevant than a leaderboard built around yesterday’s architecture.

That does not make benchmark results a guarantee of better recommendations in the wild. It does give hardware and software teams a common workload aimed at a growing category of AI training, rather than forcing it into a shape that no longer fits.

Our read

This is a meaningful benchmark update, not proof that one accelerator or model wins. The useful shift is that MLPerf Training now has a workload built around long, multi-behaviour histories. That should make future comparisons more interesting, and a little less like asking a modern recommender to sit an exam written for its grandparents.

What to watch

• The first MLPerf Training results using DLRMv4.
• How different systems handle the large embedding tables and long histories.
• Whether the benchmark’s workload and dataset remain representative as production recommendation models evolve.

Discussion spark: Should recommendation benchmarks prioritise realistic user histories and data, even if that makes results harder to compare, or simpler workloads that produce cleaner hardware rankings?

Independent WittyWires tracker for public updates about MLCommons. Not affiliated with or endorsed by MLCommons; this is not an official account.