AWS and Qumulo say they have trained a large language model on a SageMaker HyperPod cluster in Oregon while its dataset stayed in Ohio, matching the performance of a cluster beside the data after an initial cache warm-up. For teams weighing expensive data replication against slower remote reads, that is a concrete alternative worth examining, with a vendor-run test rather than a general performance guarantee.
AWS AI Watch analysis
What happened
The AWS walkthrough describes a setup using SageMaker HyperPod, Qumulo’s Cloud Native Qumulo storage and Cloud Data Fabric. A Qumulo hub in AWS’s Ohio region holds the dataset; a spoke in Oregon presents it to the training cluster through a local file-system mount, fetching and caching data across the regions rather than requiring a full copy beside the compute.
AWS says it tested the approach with a 1.02-billion-parameter LLaMA v3 training job, using two p5.48xlarge instances in each cluster and 16 H100 GPUs in total. In its reported run, the local cluster processed 116–117 samples per second. The remote cluster reached 115–116 samples per second after its cache warmed. On a cold start, performance was lower for the first 100–150 batches before converging. AWS says subsequent warm-cache runs served 94–96% of reads from local NVMe at under 5 ms latency.
Why it matters
Training data and available GPU capacity do not always live in the same region. Copying very large datasets can add cost and operational work; reading everything over a long-distance link can slow training. This design aims to keep one authoritative dataset while caching the pieces a training job is expected to need near the compute.
The useful detail is the warm-up, not just the near-matching throughput. Teams considering this approach would need to account for that initial slowdown, and the reported results come from AWS and Qumulo’s particular configuration, not a promise that every dataset, network or workload will behave the same way.
Our read
This is a practical infrastructure story: less time shuffling petabytes around, potentially more flexibility about where scarce accelerators run. The test makes a persuasive case for trying predictive caching on workloads with repeatable data access. It does not make cross-region training magically local, and the first batches still pay the distance tax. If your data loader’s habits are less predictable, the cache may have a less impressive day than the demo.
What to watch
- Whether AWS or Qumulo publish results for other datasets, model sizes and data-loading patterns.
- How much the cold-start penalty changes with longer jobs or less predictable reads.
- The full cost and operational trade-offs against copying data or placing compute beside it.
Discussion spark: Would you trust predictive caching to keep a long AI training run fed across regions, or is copying the dataset still the safer bet?
Sources and evidence
- Multi-Region training with Amazon SageMaker HyperPod and Qumulo (25 September 2026, 15:49 UTC)
not affiliated with or endorsed by Amazon Web Services (AWS)