Verda’s data-streaming test lifts GPU training throughput nearly sixfold

Verda says streaming training data from object storage to local NVMe substantially increased throughput in its test, reaching nearly six times the baseline once the cache was warm. The result shows why AI training performance can depend as much on feeding GPUs as on adding more of them.

Verda Watch analysis

What happened

Verda compared a standard PyTorch DataLoader using Parquet files with MosaicML Streaming, or MDS, for a ResNet-18 image-classification workload. The test used 500,000 images from the iNaturalist 2021 mini dataset and measured 204,800 sample presentations on two nodes with 16 NVIDIA B300 GPUs in total.

The company reports throughput of 2,638 images per second for the DataLoader baseline, 9,970.7 for streaming on a cold cache and 15,809 on a warm cache. Measured wall time fell from 77.63 seconds for the baseline to 20.54 seconds cold and 12.95 seconds warm. Verda says the exercise measured system performance, not whether the model learned to convergence.

Why it matters

Training accelerators can sit idle when the next batch of data arrives too slowly. Verda’s comparison suggests that sharding data, streaming it asynchronously and caching it locally can help keep a large training setup busy. The warm-cache result is especially striking, though it comes with a practical detail: the first run took longer to get its first batch than the baseline.

The measurements come from a small, 200-step experiment on Verda’s own infrastructure, so they are not a universal promise of sixfold gains. Workloads that already fit comfortably on local SSD or NVMe may have little to gain from a more elaborate pipeline.

Our read

This is a useful, concrete case for testing the data pipeline before reaching for more GPUs. The headline improvement is worth attention, but the meaningful comparison is the one on your own workload, including the cost and delay of the cold start. Verda’s guide also gives operators practical signals to check: input-wait time, GPU utilisation and throughput, rather than treating one dashboard number as the whole story.

What to watch

• Whether similar gains hold on larger datasets and different model workloads.
• How much setup, storage and network cost the streaming approach adds.
• Whether cold-start delays matter for jobs that run only briefly.

Discussion spark: When a training run is bottlenecked by data loading, should teams first invest in better pipelines and caching, or is adding more GPU capacity still the more practical fix?

not affiliated with or endorsed by Verda