NVIDIA’s Dynamo splits multimodal AI serving into three faster-moving parts

Multimodal AI serving is getting a plumbing upgrade: NVIDIA’s Dynamo framework can split vision encoding from language-model prefill and decode, allowing each stage to batch, schedule and scale independently. NVIDIA says the approach is most useful for image-heavy requests, shorter outputs and mixed traffic where image jobs can otherwise hold up text requests.

NVIDIA Watch analysis

What happened

In an official technical write-up published on 9 September, NVIDIA describes three deployment patterns: keep everything together, place encoder workers alongside the language-model workers, or run encoders on a separate, lower-cost GPU tier. The resulting image embeddings are passed to the prefill-and-decode workers through NVIDIA’s Inference Transfer Library.

The headline results come from NVIDIA’s own tests using Qwen3.5 configurations. For a ten-image workload, colocated encoder workers cut time to first token by 58%, while a heterogeneous setup cut it by 50%. In mixed text-and-image traffic, NVIDIA measured mean first-token latency falling 42.2% for text requests and 30.8% for image requests.

Key findings

• Separate the vision bottleneck
  Encoding images no longer has to queue inside the same worker as language-model prefill and decode.
• Text requests can move sooner
  NVIDIA measured mean text-request time to first token falling from 92.3 ms to 53.3 ms in mixed traffic.
• Heterogeneous hardware has a role
  A lower-cost GPU tier can handle encoder work while premium GPUs focus on the language model.
• Long answers blunt the advantage
  When decode dominates, separating the encoder improves first-token latency more than total response time.
• Bigger models change the maths
  NVIDIA found the benefit can fall as the language model becomes a larger share of total work.

Why it matters

This is a useful shift in how teams think about multimodal inference. The vision encoder is not merely a preliminary step to hide inside one giant serving worker. For the right workload, it becomes an independently manageable service, which could mean better responsiveness without buying more top-tier GPUs for every stage.

The catch is workload fit. NVIDIA’s strongest numbers come from specific models, traffic patterns, image counts, latency targets and hardware. Treat them as a design signal, not a universal speed-up button.

Our read

Dynamo’s interesting contribution is architectural rather than theatrical: it makes the awkward front end of multimodal AI visible and tunable. Teams serving lots of images should model encoder time separately from prefill and decode, then test whether colocation or a cheaper encoder tier pays off.

What to watch

• Whether independent deployments reproduce NVIDIA’s latency and goodput gains.
• How the approach behaves with video, larger outputs and dense models.
• The operational cost of moving embeddings between GPU tiers.
• Which multimodal serving stacks adopt the same split-stage pattern.

Discussion spark: For production multimodal AI, would you prioritise lower first-token latency or simpler all-in-one serving, and what workload would decide it?

Independent WittyWires tracker for public updates about NVIDIA. Not affiliated with or endorsed by NVIDIA; this is not an official account.