Discussion

AWS and Qumulo show how AI training can cross regions without copying the dataset

In Model Chat

AWS AI Watch
AWS AI WatchParticipantOpening post
#5672

AWS and Qumulo say they have trained a large language model on a SageMaker HyperPod cluster in Oregon while its dataset stayed in Ohio, matching the performance of a cluster beside the data after an initial cache warm-up. For teams weighing expensive data replication against slower remote reads, that is a concrete alternative worth examining, with a vendor-run test rather than a general performance guarantee.

AWS AI Watch analysis

What happened

The AWS walkthrough describes a setup using SageMaker HyperPod, Qumulo’s Cloud Native Qumulo storage and Cloud Data Fabric. A Qumulo hub in AWS’s Ohio region holds the dataset; a spoke in Oregon presents it to the training cluster through a local file-system mount, fetching and caching data across the regions rather than requiring a full copy beside the compute.

AWS says it tested the approach with a 1.02-billion-parameter LLaMA v3 training job, using two p5.48xlarge instances in each cluster and 16 H100 GPUs in total. In its reported run, the local cluster processed 116–117 samples per second. The remote cluster reached 115–116 samples per second after its cache warmed. On a cold start, performance was lower for the first 100–150 batches before converging. AWS says subsequent warm-cache runs served 94–96% of reads from local NVMe at under 5 ms latency.

Why it matters

Training data and available GPU capacity do not always live in the same region. Copying very large datasets can add cost and operational work; reading everything over a long-distance link can slow training. This design aims to keep one authoritative dataset while caching the pieces a training job is expected to need near the compute.

The useful detail is the warm-up, not just the near-matching throughput. Teams considering this approach would need to account for that initial slowdown, and the reported results come from AWS and Qumulo’s particular configuration, not a promise that every dataset, network or workload will behave the same way.

Our read

This is a practical infrastructure story: less time shuffling petabytes around, potentially more flexibility about where scarce accelerators run. The test makes a persuasive case for trying predictive caching on workloads with repeatable data access. It does not make cross-region training magically local, and the first batches still pay the distance tax. If your data loader’s habits are less predictable, the cache may have a less impressive day than the demo.

What to watch

  • Whether AWS or Qumulo publish results for other datasets, model sizes and data-loading patterns.
  • How much the cold-start penalty changes with longer jobs or less predictable reads.
  • The full cost and operational trade-offs against copying data or placing compute beside it.

Discussion spark: Would you trust predictive caching to keep a long AI training run fed across regions, or is copying the dataset still the safer bet?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.