vLLM 0.34.0 brings 10 new models and makes extensible KV-cache sizing the default, with the project reporting higher throughput for most models. The substantial serving release also removes Model Runner V1 and changes defaults that operators should check before upgrading.
vLLM Watch analysis
What happened
The vLLM project published version 0.34.0 on 16 November. Its release notes say the engine now sizes the KV cache using memory actually free after warm-up and CUDA graph capture, rather than an estimate. The notes give MiniMax-M3 on H200 as an example: cache capacity rises from 433,000 to 672,000 tokens.
The release adds 10 models, led by Mistral Large 4, and makes Model Runner V2 the only GPU runner. Other changes include checkpoint and restore endpoints, fault-tolerant scale-down for distributed deployments, and new serving and speculative-decoding options.
Our top picks
- Extensible KV cache, on by default
More available memory can mean a larger cache, higher throughput and lower time to first token at high concurrency. - Ten new models
Additions include Mistral Large 4, MiniMax-M3.1-Flash and NVIDIA Nemotron-3.5-Transcribe. - Checkpoint and restore endpoints
Operators can prepare, check and restore checkpoints through the Python and Rust frontends. - Fault-tolerant scale-down
Distributed deployments can remove dead ranks and redistribute experts without restarting. - A single GPU runner
Model Runner V1 is removed, leaving Model Runner V2 as the only GPU runner.
Why it matters
This release gives serving teams practical changes to model choice, memory use and recovery. The cache change could improve capacity and throughput, but the notes also warn that per-token decode latency can rise by 6–18% on some H200 models at concurrency 128. Results will depend on the workload, not merely the optimism of the upgrade command.
There are compatibility changes to plan for: the default block size rises from 16 to 64, ROCm 7.2 artifacts are dropped, and huggingfacehub 2.0 or later is required. Teams can retain the old block size with –block-size 16; the release notes list additional migration steps.
Our read
This is a meaningful infrastructure release, not a changelog whose main achievement is being long. The cache and recovery changes stand out, but the altered defaults make a staged test the sensible first move. vLLM’s performance figures are project-reported results, not a guarantee for every deployment.
What to watch
- Whether the larger cache improves throughput on operators’ own workloads.
- How the default block-size change affects existing deployments.
- Whether teams encounter latency trade-offs or migration issues after upgrading.
Discussion spark: Would you enable vLLM’s larger default KV cache for the potential throughput gains, or keep the old sizing until your own workload confirms the trade-off?
Sources and evidence
- Releases · vllm-project/vllm · GitHub (Publication date not supplied)
This is an independent WittyWires tracker and is not operated by, endorsed by, or affiliated with vLLM or the vLLM Project.