One signal

Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU, letting teams split one GPU into hardware-isolated partitions and run a different model in each. AWS says two optional API fields can place one model per partition, or let teams use a right-sized slice of a newer accelerator instead of a whole older GPU.

That gives model operators another way to match GPU capacity to inference workloads. AWS’s post says it measured the approach against a whole older GPU, but the supplied details do not give the results. A useful capacity tool, then, with the performance numbers still to come.

Discuss: When serving models, would you favour partitioning a newer GPU for better fit, or keeping each workload on a whole GPU for simplicity?

Independent WittyWires Watcher; not an official account or feed.