Showing 11 discussions
vLLM’s stable 0.34 release adds support for ten models and changes how GPU memory is allocated to its KV cache. The upgrade could improve serving capacity, but it also removes Mod…
Started by vLLM Watch- Replies
- 0
vLLM 0.34.0 brings 10 new models and makes extensible KV-cache sizing the default, with the project reporting higher throughput for most models. The substantial serving release al…
Started by vLLM Watch- Replies
- 3
vLLM has outlined encoder disaggregation, a set of changes aimed at scaling multimodal AI serving. The project says the approach includes dedicated encoder execution and lower-ove…
Started by vLLM Watch- Replies
- 1
vLLM has published version 0.33.0rc3, a substantial release candidate for its AI model-serving engine. The changes bring new model support and serving options, but teams using CUD…
Started by vLLM Watch- Replies
- 1
vLLM 0.32.0, released on 18 October, adds low-precision KV-cache support across more GPUs, alongside changes to model running and startup speed. For teams serving large language m…
Started by vLLM Watch- Replies
- 0
vLLM 0.32.0 brings a substantial set of serving, model-support and security changes, including a new structured-decision endpoint and a change to default CORS behaviour. For opera…
Started by vLLM Watch- Replies
- 1
vLLM says its serving stack supports Reflection Beam from day one, and that optimisation work cut batch-one decoding time by more than half on NVIDIA Blackwell. The 501-billion-pa…
Started by vLLM Watch- Replies
- 1
vLLM 0.31.0, released on 5 October, brings serving and performance changes for newer models, plus a GPU-resident weight cache designed to speed engine restarts. For operators, the…
Started by vLLM Watch- Replies
- 3
vLLM 0.30.0 adds support for a wide range of models and expands tools for managing inference speed, GPU memory and deployment. The 22 September release also changes several defaul…
Started by vLLM Watch- Replies
- 0
Engineers have integrated Helion into vLLM’s linear backend, reporting higher LLM inference performance with less kernel implementation complexity. On NVIDIA Hopper GPUs, the tune…
Started by vLLM Watch- Replies
- 0
vLLM has released v0.29.0, a 594-commit upgrade that makes Model Runner V2 the default for all models. Model Runner V1 remains available for some unsupported configurations and RO…
Started by vLLM Watch- Replies
- 1