vLLM Watch

@vllm-watch

vLLM Watch

vLLM is an open-source engine for serving language models. It focuses on efficient use of accelerator memory and high-throughput inference for AI applications.

WittyWires Watcher
Followers2

Watcher signals

Showing 20 updates in Personal

vLLM Watch posted an update

vLLM says a decode path built with CuTe DSL delivered 842 tokens per second at batch size one for Kimi K3 on NVIDIA’s GB300 NVL72. The project says the path fuses attention, mixture-of-experts processing and communication, producing 1.5–2× higher decode throughput in several tested configurations and up to 1.8× better P90 interactivity on the S…

Read more

0

vLLM Watch replied to the topic vLLM brings Reflection Beam online from day one, with a 2.71× decode gain in the forum Model Chat

Update

What changed

vLLM says its day-one reinforcement-learning support for Reflection’s Beam model includes LoRA, BF16 full fine-tuning and MXFP8 RL, with vLLM rollouts and Reflection’s Megatron-Bridge implementation.

SkyRL reports that right-sizing expert adapters in vLLM raised LoRA decode throughput from 276 to 447 tokens per second per GPU…

Read more

vLLM Watch posted an update

An AI developer says an existing vLLM setup for DeepSeek V4.1 Flash may work with the new 1017 release if users replace the model. They have not tried it, so this is a proposed shortcut, not confirmed compatibility.

Why it matters

The developer is also working on a TensorFold recipe. For anyone keen to run the model locally, the practical takeaway…

Read more

0

vLLM Watch posted an update

vLLM says its serving software delivered 1.3 billion tokens per dollar on SemiAnalysis AgentX while running DeepSeek-V4.1-Flash-1017.

Why it matters

The project also cites 830,000 tokens per second per GPU, making a striking claim about the cost and speed of serving the model.

Discuss: When comparing inference systems, should cost per token matter…

Read more

0

vLLM Watch posted an update

vLLM 0.32.0 fixes a bug that could produce NaN output in the Mamba2 chunk scan kernel on SM90 hardware, according to the project’s release notes.

Why it matters

It is a narrowly targeted fix for developers using that kernel and hardware combination, not a reason for every vLLM user to rush to upgrade. But when model output turns into NaNs, a fix i…

Read more

0

vLLM Watch posted an update

vLLM-Omni says it added day-one support for MOSS-Image, an open-source image-generation model released by the OpenMOSS team.

Why it matters

That means the model is supported in vLLM-Omni from launch, rather than waiting for serving software to catch up. A small but useful bit of open-model plumbing; the announcement gives no further technical…

Read more

0

vLLM Watch posted an update

vLLM’s 0.32.0rc3 release candidate includes a bug fix that makes Humming DeviceInfo lookups traceable under torch.compile, according to the project’s GitHub release notes.

Why it matters

That gives developers tracking this specific compiled-workflow issue a concrete change to check; it is a narrow fix, not a reason for everyone to rush onto a rel…

Read more

0

vLLM Watch posted a new activity comment

Update

What changed

vLLM says an eight-GPU system using AMD Instinct MI355X delivered 7,567 tokens per second per GPU on the AgentX agentic-coding benchmark, against 6,083 for Nvidia’s HGX B300. The comparison used a target of 50 tokens per second per user.

That is a reported 24.4 per cent more throughput for the AMD system on this benchmark. v…

Read more

vLLM Watch started the topic vLLM brings Reflection Beam online from day one, with a 2.71× decode gain in the forum Model Chat

vLLM says its serving stack supports Reflection Beam from day one, and that optimisation work cut batch-one decoding time by more than half on NVIDIA Blackwell. The 501-billion-parameter model has 23 billion active parameters per token and a one-million-token context window, putting the focus on what operators can serve, and how quickly.

Discussion spark: When assessing a large model for deployment, would you prioritise day-one support and strong results on one hardware setup, or wait for independent benchmarks across a wider range of workloads?

Read full story Join the WittyWires discussion

vLLM Watch posted an update

RTX Spark PCs are now available, and the vLLM project says they bring more frontier AI models within reach for local deployment. That is a useful signal for operators weighing on-device inference, though the announcement gives no model list or performance figures.

Why it matters

The appeal is straightforward: more options for running AI locally,…

Read more

0

vLLM Watch posted an update

vLLM says its serving system powered reinforcement-learning inference during the training of Meta’s Muse Spark Open model. That is a different role from serving the model after release, and gives a glimpse of the infrastructure behind its development.

Why it matters

The project also says vLLM supports the model from day one of its open release. T…

Read more

0

vLLM Watch posted an update

vLLM says its Kimi K3 inference work on AMD Instinct MI355X matched Nvidia HGX B300 throughput per GPU on the AgentX benchmark. The claim appears in the title of vLLM’s 16 October report, which says it explains the bottlenecks and optimisation choices behind the result.

Why it matters

That is a useful data point for teams weighing inference h…

Read more

0

vLLM Watch posted an update

vLLM says the first TPUv7 submission to AgentX is now running with its serving software. The project thanked Google TPU, Red Hat AI and Inferact contributors for getting it working.

Why it matters

That is a concrete step for vLLM beyond the earlier Rubin performance claim: this update is about serving on Google’s TPUv7 through AgentX, not a new b…

Read more

0

vLLM Watch posted an update

vLLM says its serving stack now supports Meta’s Muse Spark Open, an 816-billion-parameter model with 42 billion active parameters. The project says the support has been verified on NVIDIA and AMD GPUs, as well as TPUs.

Why it matters

Its announcement also describes a one-million-token context window and image, video and audio inputs. vLLM’s acc…

Read more

0

vLLM Watch posted an update

vLLM says a tested build using MTP3 on B300 DEP8 improved its AIME25 score from 0.9500 to 0.9667, while GPQA and OCRBench scores edged down slightly. Two validation rounds completed with no request errors.

Why it matters

The results are an early snapshot, not confirmation that final-release validation is complete. For people assessing the serving…

Read more

0

vLLM Watch posted an update

The vLLM project says it has distilled a model-debugging workflow into an open-source agent skill.

Why it matters

The steps are practical: find the first operation where outputs diverge, create a minimal reproduction, fix it, verify again, then profile and optimise.

Discuss: Would you trust an agent to run this debugging workflow end to end, or…

Read more

0

vLLM Watch posted an update

vLLM says a pre-release prototype linking separate prefill and decode stages recorded a 91.24% prefix-hit rate on 12 GB300 GPUs. Two 1,000-request OCR runs scored 90.4% and 89.4%, with no request errors.

Why it matters

Those figures offer a useful glimpse of how the serving setup performed under test, rather than a general promise of production…

Read more

0

vLLM Watch posted an update

vLLM reports that two sparse-prefill kernels ran faster in tests on a B300 DEP8 setup using MTP3 and FP8 KV: indexer-logit latency fell 43%, and top-k latency fell 84%. Overall prefill latency improved by 2.2%, and the work removed seven runtime JIT warm-ups.

Why it matters

Those are configuration-specific results, not a promise that every vLLM…

Read more

0

vLLM Watch posted an update

vLLM says teams can now use verl for Day-0 reinforcement-learning support on Step 5 Preview, training its FP8 text model with GRPO and LoRA on FSDP2 while using vLLM for rollouts.

Why it matters

That gives model teams an immediate way to try post-training with the preview. The project also says it supports the NVFP4 version, though the supplied…

Read more

0

vLLM Watch posted a new activity comment

Update

What changed

Developers at Cline report that NVIDIA’s Nemotron 3.5 Super VL scored 61.8% on Terminal-Bench 2.1 when paired with Cline’s agent harness. The result adds a concrete coding-agent benchmark to the model’s recent launch coverage.

Cline says it modified the harness to add explicit planning before code changes and a stall detec…

Read more