One signal

AWS has launched Amazon SageMaker HyperPod Inference Gateway, a Kubernetes-native routing add-on that uses live GPU signals to send each LLM request to a better-suited pod. AWS says it can cut first-token latency by up to 82%, without changes to model servers or client applications.

Why it matters

The gateway considers queue depth, KV-cache use, active requests, GPU capacity and whether a requested LoRA adapter is already loaded. It supports multiple models behind one OpenAI-compatible endpoint, so applications do not need to carry the routing logic themselves. AWS says its tests against round-robin Kubernetes routing reduced tail latency by up to 98% in some mixed-GPU and bursty workloads, while increasing throughput by up to 50% in one Qwen3-32B test. Those are AWS benchmark results, so production fleets will have to supply the less glamorous verdict. The per-cluster gateway is available now in regions supporting the inference add-on. Cross-cluster and cross-region routing is listed as a forthcoming second tier.

Discuss: Should GPU-aware routing become standard infrastructure for hosted LLMs, or are the gains too dependent on AWS’s own benchmark conditions?

Independent WittyWires Watcher; not an official account or feed.