vLLM adds two-way KV-cache transfer to speed up long agent conversations

vLLM has added bi-directional KV-cache transfer for multi-turn AI workloads, letting prefill servers reuse context held by decode servers instead of repeatedly computing it. In AWS’s benchmark, the change cut time to first token by up to 3.3 times for one tested model, though the gains depend on workload and configuration.

vLLM Watch analysis

What happened

The vLLM post, written by AWS’s Sunita Nadampalli and published on 12 October, describes a change to disaggregated serving, where separate nodes handle prompt processing and token generation. Previously, KV-cache data moved from prefill to decode, but not back again. With the new transfer, decode nodes can return eligible cached context to prefill nodes on later turns, reducing repeated computation as a conversation grows.

The setup uses a KV-cache-aware router to track where that data is held. The router carries metadata, while KV tensors move directly between nodes. Decode nodes retain eligible cache blocks for a configurable period, 480 seconds by default, and a recompute threshold determines when fetching cached data is worthwhile rather than calculating it again.

AWS tested the feature on a one-prefill, one-decode deployment using EC2 P5en instances connected by Elastic Fabric Adapter. It reports up to 1.6 times lower time to first token for Qwen3-32B and up to 3.3 times lower for GLM-5-FP8 in its tested long-prompt cases. AWS also says the results vary with workload and configuration.

Why it matters

Agents that call tools and build up conversation history can make servers redo substantial prompt-processing work on each turn. Reusing cached context could mean lower latency and less accelerator capacity spent on repeated computation, particularly as interactions lengthen.

The practical limits matter: cache retention uses accelerator memory, the default expiry is fixed rather than renewed, and transferring a short cache can cost more than recomputing it. AWS’s results are from a specific benchmark setup, not a guarantee that every deployment will see the same improvement.

Our read

This is a concrete serving improvement with a clear target: long, context-heavy conversations where repeated prefill work becomes a drag. The best part is that the post explains the operational knobs, not just the headline speed-up. Teams should compare the benefit against their own interconnect, model and conversation patterns before treating 3.3 times as a budget line.

What to watch

• Whether independent workloads reproduce the reported latency gains.
• How deployments tune cache retention and the recompute threshold.
• Whether the router and connector approach is adopted across more serving setups.

Discussion spark: For long-running AI agents, would you prioritise reusing cached context even when it consumes more accelerator memory, or recomputing it to keep the serving system simpler?

This is an independent WittyWires tracker and is not operated by, endorsed by, or affiliated with vLLM or the vLLM Project.