vLLM adds bi-directional KV reuse to speed up long agent conversations

vLLM has added bi-directional KV-cache transfer for multi-turn workloads, letting a prefill server reuse context held by a decode server instead of recalculating it. In AWS’s tests, that cut time to first token by up to 1.6 times for Qwen3-32B and 3.3 times for GLM-5-FP8 at the longest prompts tested.

vLLM Watch analysis

What happened

The feature is for deployments that split prefill, which processes the prompt, from decode, which generates the answer. On later turns, decode can return cached prompt and response context to prefill, reducing repeated computation as an agent’s conversation grows.

AWS says the benchmarks used one prefill and one decode node on EC2 P5en instances connected by Elastic Fabric Adapter. Results vary with workload and configuration. The feature also needs a KV-cache-aware router to coordinate reuse across turns.

Why it matters

Long conversations can mean more repeated prefill work, higher latency and extra accelerator use. Reusing cached context could help operators keep responses snappy as conversations grow, rather than paying to re-read the same history turn after turn.

There are knobs to mind: cached blocks are retained for a configurable lifetime, set to 480 seconds by default, and a default threshold of 64 tokens determines when reuse is preferred over recomputation. That gives operators control, but also makes the benefit dependent on tuning and deployment design.

Our read

This is a practical infrastructure improvement, not a promise that every agent gets 3.3 times faster. The headline gains come from AWS’s specific benchmarks, and the source says results vary. Still, avoiding duplicated work in long-running conversations is exactly the sort of unglamorous engineering that can make agent systems more usable at scale. Operators should compare the feature against their own models, traffic and network setup before celebrating the benchmark in production.

What to watch

• Whether the gains hold across other models, hardware and traffic patterns.
• How operators tune cache retention against accelerator-memory use.
• Whether router support and deployment tooling make multi-turn reuse straightforward.

Discussion spark: For agent workloads, would you prioritise reusing cached context to cut latency, or keeping the serving architecture simpler even if it means recalculating more?

This is an independent WittyWires tracker and is not operated by, endorsed by, or affiliated with vLLM or the vLLM Project.