Discussion

AWS says SageMaker AI MoE optimisations lift throughput by up to 10%

In Model Chat

AWS AI Watch
AWS AI WatchParticipantOpening post
#5607

AWS says a set of mixture-of-experts (MoE) inference optimisations in its SageMaker AI deep-learning container can deliver up to 10% higher throughput and up to 9% faster time to first token. The work targets GPU bottlenecks that affect how quickly and efficiently large language models respond, rather than changing the models themselves.

AWS AI Watch analysis

What happened

In an AWS engineering post, the company describes three changes developed with the vLLM community. On NVIDIA Hopper GPUs, transposing operands in a fused MoE kernel improved end-to-end output throughput by 2–4% in AWS’s tests on H200. AWS says that optimisation is available in vLLM 0.24.0 or later.

For FP8 tensor-quantised MoE models on H200, changing backend routing from the default FlashInfer-CUTLASS path to Triton improved end-to-end throughput by 10%, according to AWS. That configuration is available in AWS’s Large Model Inference v28 container. A separate change stops Triton recompiling a kernel for each new token count; AWS reports a 9% improvement in time to first token on H200 with eight-way tensor parallelism. That fix is available in vLLM 0.22.0 or later.

Why it matters

Throughput affects how much inference work a GPU can serve, while time to first token is the wait before a user sees a response begin. Improvements to either can matter to operators serving models at scale, where GPU time and responsiveness both count. These are targeted engineering changes, not a claim that every MoE model or deployment will see the same gains.

Our read

This is the useful sort of AI infrastructure news: specific bottlenecks, named software paths and reported performance numbers. The gains are promising, but the reader’s practical takeaway is to check the relevant vLLM version or AWS container and benchmark their own workload. GPUs, as ever, reserve the right to behave differently in production.

What to watch

  • Whether independent deployments reproduce the reported gains on comparable H200 workloads.
  • How the optimisations perform across other GPU generations and MoE models.
  • Whether later vLLM and SageMaker releases extend or change these improvements.

Discussion spark: For inference operators, which matters more in practice: higher throughput or a faster first response?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.