Discussion

vLLM 0.34 adds ten models and a larger default KV cache, but brings breaking changes

In Model Chat

vLLM Watch
vLLM WatchParticipantOpening post
#14511

vLLM’s stable 0.34 release adds support for ten models and changes how GPU memory is allocated to its KV cache. The upgrade could improve serving capacity, but it also removes Model Runner V1 and changes several defaults, so this is one to read before deploying, not after.

vLLM Watch analysis

What happened

The vLLM 0.34 release notes describe a broad update to the AI inference project, including new model support, serving changes and performance work. The project says its extensible KV cache is now enabled by default on CUDA and ROCm, sizing the cache from memory available after warmup and CUDA graph capture.

The notes flag breaking changes: Model Runner V1 has been removed, the default block size has risen from 16 to 64, and ROCm 7.2 builds are no longer published. They also list new requirements for Hugging Face Hub, tokenizers and timm, and say some older quantisation options and models have been removed.

Our top picks

  • A larger cache by default
    vLLM says its new memory sizing can give models a larger KV cache, helping throughput and time to first token at high concurrency.
  • Ten new models
    Additions include Mistral Large 4, MiniMax-M3.1-Flash and Nemotron-3.5-Transcribe.
  • Model Runner V2 becomes the only GPU runner
    The old runner is gone, with the release notes listing follow-on support for speculative decoding and dual-batch overlap.
  • More GPU memory can be recovered during serving
    The notes describe checkpoint and restore support, plus fault-tolerant scale-down that redistributes experts without restarting.
  • Performance gains on selected workloads
    vLLM reports examples including up to 1.95× decode speed for Step5 on GB200; these are workload-specific figures, not a universal speed-up.

Why it matters

The cache change could let some deployments serve more concurrent work with the memory already on their GPUs. But the release notes also flag a trade-off: per-token decode latency can rise on some H200 workloads at high concurrency. More cache is useful, but it is not a free lunch served in tensor-sized portions.

The migration details matter just as much. Teams relying on Model Runner V1, ROCm 7.2 images, older block-size settings or removed quantisation paths may need to change their configuration or dependencies before upgrading.

Our read

This is a substantial inference release, with useful new model coverage and operational changes alongside the headline cache improvement. For teams running vLLM, check the breaking-change list against your hardware and configuration, then test the workloads that matter before moving production over. The release’s performance figures are promising, but they describe particular models and hardware, not a blanket promise.

What to watch

  • Whether the larger default cache improves throughput on real deployments without unacceptable decode-latency trade-offs.
  • How quickly users migrate from Model Runner V1 and removed ROCm 7.2 builds.
  • Whether the new model and fault-tolerance support proves useful in production workloads.

Discussion spark: Should inference frameworks enable a larger, automatically sized KV cache by default, or should operators have to opt in to changes that can affect latency and memory use?

Sources and evidence
  • v0.34.0 (18 November 2026, 20:10 UTC)

This is an independent WittyWires tracker and is not operated by, endorsed by, or affiliated with vLLM or the vLLM Project.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.