SageMaker’s inference year in review: 13 launches, two paths, one decision

Amazon has published a year-to-date accounting of its SageMaker inference work, and it reads less like a launch post than a decision guide. Thirteen capabilities have landed in 2026 across two deployment paths, and the review's real contribution is putting the choice between them on a single page.

AWS AI Watch analysis

What happened

Per AWS's year-to-date review, SageMaker AI shipped 13 inference capabilities this year along two paths: managed endpoints, where AWS handles provisioning, scaling and monitoring, and HyperPod Inference, a Kubernetes-native route for teams that want node-level control. A seven-dimension table makes the split explicit: console versus kubectl, managed autoscaling versus Karpenter and KEDA, OpenAI-compatible HTTP versus gRPC and custom load balancers.

Seven launches sit on the endpoints side, dated April to June: automated inference recommendations, capacity-aware instance pools, OpenAI-compatible APIs, container caching, token-level observability, async inline payloads and prefix-aware routing. The HyperPod side names six more, from a simplified operator to disaggregated prefill and decode, with model caching pre-loading weights and images onto cluster nodes.

Our top picks

• OpenAI-compatible APIs
  Endpoints expose an OpenAI-style path, so apps built on the OpenAI SDK or LangChain move over by changing a URL; bearer tokens replace SigV4 signing.
• Inference recommendations
  Automated benchmarking replaces two to three weeks of manual testing across 1,000-plus combinations; a demonstrated GPT-OSS-20B run doubled throughput at the same latency.
• Capacity-aware instance pools
  A prioritised list of up to five instance types lets scale-out survive a capacity shortage instead of failing before the first request.
• Container caching
  Pre-pulled images cut demonstrated startup latency from 525 to 258 seconds for a 17.7 GB serving container, a 51% reduction.
• Prefix-aware routing
  Requests sharing a prompt opening route to the same instance; KV cache hit rates rose from about 25% to 82% in AWS's Llama 3.1 70B tests.
• Token-level observability
  More than 100 inference metrics flow through OpenTelemetry into a CloudWatch dashboard, enabled by default and visible within two minutes of an endpoint going live.
• Async inline payloads
  Asynchronous requests up to 128,000 bytes travel inside the call, removing the S3 staging step and its per-invocation charge for most workloads.

Why it matters

Inference is where generative AI budgets quietly bleed: cold starts measured in minutes, capacity shortages that kill endpoints before their first request, and token-level signals that traditional monitoring never surfaced. The review is candid about that, and its two-path framing turns a year of scattered launch notes into an architecture decision a team can actually make.

The quietly consequential entry is the OpenAI-compatible API. Applications built on the OpenAI SDK, LangChain or Strands Agents migrate by changing an endpoint URL, with bearer tokens replacing SigV4 signing. That lowers the cost of trying AWS and, just as pointed, the cost of leaving it. One caveat travels with every figure here: these are AWS's own benchmarks, from the 51% startup improvement to the 82% cache hit rate.

Our read

This is the rare vendor retrospective that earns its length, because it organises the year's launches around a choice the reader genuinely has to make. Treat the numbers as AWS grading its own homework until independent runs appear, then act on the structure, which needs no benchmark to be true. In practice: managed endpoints if speed to production matters most, HyperPod if Kubernetes control is a requirement, and if your prompts share openings, PREFIXAWARE routing is a configuration change rather than a re-architecture. Prefix-aware routing and model caching already had their own launch moments this month; the review's value is the map, not the individual pins.

What to watch

• Independent numbers
  Third-party runs against the container caching and prefix-aware routing claims would turn vendor benchmarks into planning figures.
• The OpenAI-compatible surface
  Whether the 14-region footprint widens and more API paths arrive, which would test how serious the portability pitch is.
• HyperPod documentation
  Deeper guides for disaggregated prefill and decode and the tiered KV cache, two launches the review names but does not unpack.
• The cadence
  Thirteen launches by mid-September sets a pace; whether it holds to year end says much about how central inference has become to SageMaker.

Discussion spark: With SageMaker endpoints now speaking the OpenAI API, is cloud lock-in fading or just moving up a layer into the managed plumbing around the model?

not affiliated with or endorsed by Amazon Web Services (AWS)