AWS shows how to serve Qwen3.8’s 2.4T open model on eight B300 GPUs

AWS has published a deployment guide for Alibaba’s Qwen3.8-2.4T-A95B, showing how to serve the open-weight model on Amazon SageMaker HyperPod with vLLM. The practical headline is that a 2.4-trillion-parameter model can be made usable on one ml.p6-b300.48xlarge instance with eight NVIDIA B300 GPUs, using NVFP4 quantisation and an OpenAI-compatible endpoint.

AWS AI Watch analysis

What happened

The AWS Machine Learning Blog guide, published on 9 September, describes Qwen3.8-2.4T-A95B as the open-weight release of Qwen3.8-Max. AWS says the model has about 95 billion parameters active per token, native context up to 262K tokens, extensible to 1M, and a hybrid attention design intended to keep long-context serving more manageable.

The example covers cluster provisioning, model download, health checks, autoscaling and vLLM configuration. It also wires in reasoning controls, tool calling and Qwen’s built-in Multi-Token Prediction speculative decoding. This is infrastructure guidance, not an independent benchmark: the performance figures in the post are vendor-reported.

Key findings

• A huge model, compressed to fit
  NVFP4 brings the stated weight footprint to about 1.2TB, within the roughly 2.1TB of aggregate GPU memory on the eight-GPU instance.
• Long context without one giant cache bill
  Most layers use gated linear attention with a bounded recurrent state, while a smaller set uses full attention for richer token interactions.
• Agent features are part of the serving recipe
  The documented endpoint supports reasoning output, schema-constrained tool arguments and OpenAI-compatible function calling.
• The GPU capacity is not casual
  AWS says this instance is not available on demand, so deployment requires a Flexible Training Plan and matching availability-zone capacity.

Why it matters

Open weights are most useful when organisations can actually run them. AWS is packaging the decisive infrastructure pieces, from GPU allocation to node recovery, around a model aimed at coding agents, research workflows and long-horizon tool use.

The catch is wonderfully concrete: this is self-hosting with a small data-centre footprint, not a laptop download. The guide gives teams a reproducible starting point, but hardware reservation and operational cost are part of the product decision.

Our read

This is a meaningful serving milestone rather than a model-launch drum roll. If you are evaluating open-weight models for private agent workloads, the useful takeaway is the complete path from weights to endpoint. Start with the documented configuration, then measure latency, acceptance rates and cost on your own workloads.

What to watch

• Whether Qwen3.8’s long-context and tool-use behaviour holds up outside the cited benchmarks.
• How much throughput Multi-Token Prediction delivers at realistic concurrency.
• The real cost and availability of sustained B300 capacity.
• Whether other serving stacks can make the same model less hardware-hungry.

Discussion spark: For private coding and research agents, would you choose this open-weight HyperPod deployment over a hosted API once GPU reservation and operations are included?

not affiliated with or endorsed by Amazon Web Services (AWS)