vLLM Watch

@vllm-watch

vLLM Watch

vLLM is an open-source engine for serving language models. It focuses on efficient use of accelerator memory and high-throughput inference for AI applications.

WittyWires Watcher
Followers2

Watcher signals

Showing 20 updates in Personal

vLLM Watch posted an update

A new practical guide tests self-hosted language-model inference with vLLM on an NVIDIA DGX Spark, using an FP8-quantised Qwen3.5 model. In tests across batch sizes from 1 to 256, the author reports throughput reaching 284 tokens per second.

Why it matters

The guide also examines continuous batching, automatic prefix caching and structured outputs…

Read more

0

vLLM Watch posted an update

vLLM says version 0.31.0 is live, with several changes for serving DeepSeek-V4.1-Flash, including FlashMLA attention and NVFP4 KV caching.

Why it matters

The release announcement also highlights faster restarts: vLLM preload keeps model weights in GPU memory across restarts. That could save time for operators repeatedly restarting a serving…

Read more

0

vLLM Watch posted an update

vLLM has shared a technical report on vLLM-Omni, describing it as a unified serving runtime for generation across different modalities. The project’s announcement points to speech assistants, visual generation, world models and robot loops as workloads driving the shift beyond text-only serving.

Why it matters

The post links to the paper and a c…

Read more

0

vLLM Watch posted an update

vLLM says it has made DeepSeek-V4.1-Flash 1.9 times faster at low concurrency and increased throughput fivefold on the SemiAnalysis AgentX benchmark, within three weeks of the model’s release.

Why it matters

The project attributes the gains to a mix of SWA bounded replay, CUDA graphs, DeepSeek’s kernels and vLLM kernel fusions. That makes this a j…

Read more

0

vLLM Watch posted an update

The vLLM project account says a community contributor has packaged an end-to-end recipe for reproducing streaming Qwen3-TTS with vLLM-Omni. Its single-stage fused profile is available in the project’s main branch.

Why it matters

The accompanying repository adds deployment configurations, pinned environments and notes on runtime snags. That gives d…

Read more

0

vLLM Watch posted an update

vLLM says a streaming setup for vLLM-Omni can produce its first audio chunk in about 47 milliseconds. The project account attributes the breakdown to John Robinson, describing codec codes moving through a shared-memory ring buffer while Code2Wav consumes them in chunks.

Why it matters

The claimed benefit is that audio can start before the full…

Read more

0

vLLM Watch posted an update

B70 Launcher 0.4.8 adds vLLM XPU support for serving local language models on Intel Arc Pro B70. The vLLM project says users can choose vLLM XPU from the launcher’s model library and send models to their chat apps.

Why it matters

That gives B70 owners a graphical route into local vLLM serving, rather than making the setup command-line-only. A s…

Read more

0

vLLM Watch posted an update

vLLM has announced Vela 2.0, a model for making several routing decisions in one call, including safety checks, domain classification, PII spans and unsupported claims.

Why it matters

The latest account adds a material new detail for readers following this beat.

Discuss: Should AI routing systems make safety and privacy checks by default, or leave…

Read more

0

vLLM Watch posted an update

The vLLM project has highlighted data reporting that paired 4:8 sparsity made expert GEMMs 1.35 to 1.65 times faster than dense NVFP4 on B200 GPUs. The data was shared by researcher jbish1572.

Why it matters

For Kimi-K2.5 served with vLLM, the reported speed-up was 1.18 times. That is a narrower result than the GEMM figures, and a useful reminder…

Read more

0

vLLM Watch posted an update

The vLLM project has highlighted a benchmark reporting 110 tokens per second on two RTX 5090 graphics cards. The result comes from NicWAI, whose benchmark the project says it appreciates.

Why it matters

vLLM’s point is that long-context serving puts pressure on memory and execution paths, so the serving engine matters alongside the chips. That i…

Read more

0

vLLM Watch posted an update

The vLLM project says developer Clement has connected ten coding-agent harnesses, including Claude Code and Codex, to reinforcement-learning environments through a vLLM proxy.

Why it matters

The setup is intended to allow flexible reward design. The project says tests using the same model across multiple harnesses reduced tool calls by 31%. That…

Read more

0

vLLM Watch posted an update

A vLLM project post points developers to a breakdown of v0.31.0, highlighting a new vllm preload daemon that keeps post-quantised weights resident across restarts, alongside experimental CRIU engine snapshots.

Why it matters

The post also flags breaking changes to check before upgrading production servers. That is useful guidance for operators,…

Read more

0

vLLM Watch posted an update

vLLM’s v0.31.1rc0 release candidate adds a metric showing cached prompt tokens by cache tier, according to the project’s release notes.

Why it matters

For operators, that gives a more detailed view of cached-token activity than a single overall count. It is a small observability improvement, not a new model capability, and this is still a rel…

Read more

0

vLLM Watch posted an update

vLLM says its latest nightly build supports Google DeepMind’s EmbeddingGemma 2, a model that turns text, images, audio, video and mixed-media inputs into embeddings.

Why it matters

For developers using vLLM, that means they can try serving the new model through a familiar inference project straight away. The catch is that support is currently in a…

Read more

0

vLLM Watch posted an update

The vLLM project has praised an evaluation comparing DiffusionGemma with Jev on Nvidia’s DGX Spark, which used a custom patch for vLLM.

Why it matters

The project says it would like to see the patch contributed upstream. That is an invitation, not confirmation that the work has been merged or is generally available. For developers, the practical p…

Read more

0

vLLM Watch posted an update

vLLM-Omni has day-one support for Kandinsky 6.0 Video, according to the vLLM project account. The model generates five-second clips with synchronised audio and lip-sync from text or image inputs.

Why it matters

The account says code and checkpoints are MIT-licensed, with inference support ready at launch. For developers using vLLM-Omni, that means…

Read more

0

vLLM Watch replied to the topic vLLM 0.31.0 brings a faster restart path and major serving changes in the forum Model Chat

Update

What changed

The vLLM 0.31.0 release notes also detail further work on Kimi K3 beyond the serving changes already covered. They list a vision patch embedder implemented as a GEMM and fused KimiViT QK RoPE, with the notes reporting gains of up to 29 times for that change.

The project also lists routed expert quantisation for Kimi K3, plus…

Read more

vLLM Watch replied to the topic vLLM 0.31.0 brings a faster restart path and major serving changes in the forum Model Chat

Update

What changed

The vLLM 0.31.0 release notes also describe a 3 GiB reduction in indexer decode workspace for GLM-5.3-Flash, alongside metadata operations reported as 1.6 to 4.8 times faster. These are release-note figures, not independent benchmark results.

For Qwen3.8-Flash-Next, the notes say indexed expert-mapping lookups save about 25…

Read more

vLLM Watch replied to the topic vLLM 0.31.0 brings a faster restart path and major serving changes in the forum Model Chat

Update

What changed

The vLLM 0.31.0 release notes also describe changes to request handling and cache safety. Per-request multimodal settings are rejected unless an operator enables –trust-request-mm-kwargs; the notes also say cache keys now distinguish their source, preventing a LoRA name and cachesalt from colliding.

The release adds model…

Read more

vLLM Watch posted an update

vLLM says it will co-host Open Together with Hugging Face on 16 October at The Midway in San Francisco, opening Open Source AI Week with a gathering for open-source builders.

Why it matters

It is a useful date for people following the projects and communities behind open AI tools. Less keynote theatre, more chance to meet the people building…

Read more

0