AWS says its EKS setup delivers 40% more throughput for large-scale MoE training

AWS has published an architecture for training mixture-of-experts models with reinforcement learning on Amazon EKS, claiming 40% more rollout-generation throughput by combining EFA networking with DeepEP. The practical point is that scaling AI training is increasingly a traffic-management problem, not simply a hunt for more GPUs.

AWS AI Watch analysis

What happened

In a post published on 25 September, AWS describes a system for large reinforcement-learning workloads built from Amazon EKS, Elastic Fabric Adapter, DeepEP and Amazon S3. The setup separates rollout generation, policy training, reward-model inference, environment work and checkpoint storage across different worker groups.

The architecture is aimed at MoE models, which activate only selected expert networks for each token. That can reduce inference work, but training introduces heavy, irregular all-to-all communication as tokens move between experts on different accelerators. AWS says DeepEP’s specialised dispatch and combine kernels, running over EFA, reduce that communication burden.

AWS reports 40% higher rollout-generation throughput in its example, alongside figures including 2.31 times higher throughput in one comparison and a 1.31 times improvement in another. Those measurements belong to AWS’s own described configuration, not an independent benchmark.

Key findings

• Separate worker groups
  Rollout generation, policy training, reward models and preprocessing can scale independently rather than competing for one pool of hardware.
• DeepEP over EFA
  Specialised kernels route sparse expert traffic across nodes while using NVLink inside an instance.
• EKS orchestration
  Kubernetes handles placement, scaling, recovery and coordination across the different workloads.
• S3 checkpoints
  Training artefacts and model weights are stored separately from the high-frequency experience buffer.

Why it matters

Large-scale reinforcement learning runs two very different workloads at once. Rollout workers need high aggregate inference throughput, while policy-training workers need tightly synchronised progress. If either side falls behind, expensive accelerators sit idle or the training job stalls.

The AWS design matters because it treats networking and scheduling as part of the model-training system. For builders, the useful lesson is concrete: keep rollout and policy-training capacity distinct, use high-bandwidth links for inter-node expert traffic, and place durable checkpoints off the critical path. More GPUs alone may simply create a larger, more expensive traffic jam.

Our read

This is a substantial infrastructure report rather than a shiny new model announcement. AWS is making the unglamorous case that the next gains in AI development may come from moving tokens more intelligently, not merely adding silicon and hoping the cables remain emotionally supportive.

The 40% figure is worth noting, but not engraving on a server rack. It is a company-reported result tied to AWS’s hardware, software and workload choices. The more durable idea is the architecture: MoE training needs specialised communication, and reinforcement learning needs different parts of the system to stay in step.

What to watch

• Whether AWS publishes the full workload, hardware configuration and baseline behind its throughput figures.
• Whether DeepEP over EFA improves end-to-end training time, not only rollout generation.
• How the design behaves when expert traffic is uneven or worker groups scale at different rates.
• Whether comparable architectures appear on other cloud and networking stacks.

Discussion spark: Should the next big efficiency gains in AI training come from better models, or from better orchestration of the hardware underneath them?

not affiliated with or endorsed by Amazon Web Services (AWS)