AWS SDK adds hardware-isolated GPU partitions for running multiple models

AWS has added support for splitting NVIDIA GPUs into hardware-isolated partitions for SageMaker inference workloads, so teams can run multiple models on one GPU. The change gives AI operators a new way to share accelerator capacity without treating isolation as an optional extra.

AWS AI Watch analysis

What happened

The capability appears in AWS SDK for JavaScript v3.1161.0, published on 30 October. AWS’s release notes say SageMaker endpoints can use AcceleratorPartitionLayouts on production variants, while inference components can use AcceleratorPartitionType in their compute requirements. The release notes identify P4d, P5, G7 and G7e instances as supported.

The partitions use NVIDIA Multi-Instance GPU (MIG) technology. In practical terms, a team can divide a GPU into hardware-isolated slices and assign models to them, rather than reserving an entire GPU for each one.

Why it matters

Accelerators are costly to leave idle, but sharing them without controls can create awkward questions about workload separation and resource contention. Hardware-isolated partitions offer operators a more deliberate middle ground: run multiple models on one GPU while keeping their allocated compute separate.

That could make inference deployments more flexible, especially where several models need capacity but do not each justify a whole accelerator. AWS’s notes describe the controls and supported instance families, but do not quantify utilisation gains or set out performance trade-offs. Those will depend on how teams configure their workloads.

Our read

This is a useful infrastructure change, not a promise that every GPU suddenly has room for one more model. The important detail is that AWS is exposing partition controls through the SDK for both endpoint layouts and inference components. Teams running supported instances should check the SageMaker documentation and test their own workloads before reshuffling production capacity. The GPU-sharing party is open, but the partitions have actual walls.

What to watch

• Which partition layouts AWS supports on each listed instance family.
• How teams balance model density against latency and throughput.
• Whether AWS publishes benchmarks or further guidance on allocating GPU slices.

Discussion spark: For inference workloads, would you favour hardware-isolated GPU partitions even if they limit flexibility, or prefer looser sharing to maximise utilisation?

not affiliated with or endorsed by Amazon Web Services (AWS)