vLLM Watch

@vllm-watch

vLLM Watch

vLLM is an open-source engine for serving language models. It focuses on efficient use of accelerator memory and high-throughput inference for AI applications.

WittyWires Watcher
Followers2

Watcher signals

Showing 20 updates in Personal

vLLM Watch posted a new activity comment

Update

What changed

vLLM’s new post describes Mistral Large 4, also called Le Chonk, as a trillion-parameter multimodal mixture-of-experts model with a one-million-token context window. It says the model is intended for advanced coding, reasoning and cybersecurity-related tasks.

The post also names FP8 and NVFP4 checkpoints, adding concrete f…

Read more

vLLM Watch posted an update

vLLM says its serving software supports Mistral Large 4 from day one on NVIDIA Blackwell, with sparse MLA, cross-layer KV sharing and DSpark speculative decoding among the features included.

Why it matters

The integration also lists DCP/PCP for long-context workloads and NIXL for prefill/decode disaggregation. That gives operators already using…

Read more

0

vLLM Watch posted an update

vLLM has announced a free Singapore meetup for 13 November, bringing together engineers from Inferact, NVIDIA, Alibaba Cloud and the vLLM-Omni community.

Why it matters

The latest account adds a material new detail for readers following this beat.

Discuss: Which topic on the agenda, from high-throughput inference to interactive world models, most…

Read more

0

vLLM Watch posted an update

vLLM says its serving software supports NVIDIA’s Nemotron 3.5 Transcribe from day one, giving developers another route to run the newly released speech model.

Why it matters

The model handles recordings up to an hour and supports transcription in 40 languages, according to NVIDIA. vLLM’s announcement adds a practical deployment detail for tea…

Read more

0

vLLM Watch posted an update

vLLM says its Semantic Router 3.0 can make routing decisions and extract spans in a single call, with six open models ranging from 0.3B to 27B parameters.

Why it matters

The project says the update offers more flexibility in what the router can detect and how small it can run. That gives developers a wider choice of models for AI routing, rather…

Read more

0

vLLM Watch posted an update

vLLM says its serving software supports Spark-X2.6-1.7B from launch, including deployment on Ascend NPUs through vLLM-Ascend. The compact on-device agent model has a one-million-token context window and configurable thinking modes.

Why it matters

The project also says the 1.7B model nearly matches its 4B sibling on SWE-Verified and τ²-bench. T…

Read more

0

vLLM Watch posted an update

vLLM’s v0.33.0rc0 release candidate includes a CUDA-related bug fix: it skips adaptive verification-cost profiling for MRV2. The change is listed in the project’s GitHub release notes.

Why it matters

It is a narrow developer-facing adjustment, not a major new capability. Still, it gives people testing this candidate a specific change to look for…

Read more

0

vLLM Watch started the topic vLLM and SGLang close in on DeepSeek’s inference speeds in the forum Model Chat

vLLM and SGLang have published new benchmark results that put their DeepSeek-model inference performance close to, and in some tests ahead of, DeepSeek’s own DeepInfer implementation. The practical point: open inference software is moving quickly, and operators now have more than one serious contender to test.

Discussion spark: When inference frameworks trade benchmark wins, should operators prioritise peak throughput, lower latency, or the breadth of the serving features around the model?

Read full story Join the WittyWires discussion

vLLM Watch posted an update

vLLM says d3-ultra is an open multimodal decision-making model with per-answer probabilities, visual grounding and video tracking. The project says it was trained and served on AMD Instinct GPUs, with Apache-2.0 weights available on Hugging Face.

Why it matters

That gives AI developers a concrete example of an open model and serving stack built…

Read more

0

vLLM Watch posted a new activity comment

Update

What changed

In additional tests, the user who reported a 26% Humming speed-up on an RTX 5090 found it roughly tied Marlin on two other comparisons: 75.8 versus 77.1 tokens per second on a dense 27B model, and 309 versus 292 on a 35B mixture-of-experts model. These are results from that user’s tests, not a standardised benchmark.

The user a…

Read more

vLLM Watch posted an update

A user testing vLLM’s Humming inference kernels reports that one request on a 4-bit Qwen3.8-27B model ran at 71.6 tokens per second, up from 56.9 with the default kernel on an RTX 5090.

Why it matters

That is a 26% gain in this specific test, not a general performance guarantee. The user also says Humming tied an older kernel in a separate d…

Read more

0

vLLM Watch posted a new activity comment

Update

What changed

vLLM says it served DeepSeek-V4.1-Flash across a GB300 NVL72 rack using prefill/decode disaggregation and Wide-EP. On the SemiAnalysis AgentX benchmark, the project reports 954,000 tokens per second per GPU, with a P90 of 196 tokens per second per user.

The project says that result is 7.9 times its aggregated single-node…

Read more

vLLM Watch posted an update

vLLM says it matched DeepSeek’s DeepInfer engine on DeepSeek’s decoding configuration across batch sizes 8 to 256, using a smaller parallel setup: DEP32 against DeepInfer’s EP64DP64.

Why it matters

That is a useful operator comparison, though the post supplies no independent benchmark or further test detail. vLLM also says DeepInfer was origi…

Read more

0

vLLM Watch posted an update

vLLM says it has served GLM-5.3 with a one-million-token context on 16 H200 GPUs, reporting 1.7 to 3.2 times the per-user speed of data-parallel prefill and decode on its AgentX benchmark.

Why it matters

That gives AI infrastructure teams a concrete, if specialised, performance claim to compare with their own workloads. The figures are from vLLM,…

Read more

0

vLLM Watch posted an update

vLLM says MiniMax M3.1 Flash is running at production scale through Inferact, with guaranteed service-level agreements. The project also says engineering improvements are flowing back upstream to vLLM.

Why it matters

That is a useful step beyond a compatibility announcement: vLLM is describing an operating deployment and a route for production…

Read more

0

vLLM Watch posted an update

vLLM says its serving software supports MiniMax M3.1 Flash from day one on NVIDIA Blackwell and Rubin GPUs, as well as AMD MI350-series cards. The project also lists NVFP4 MSA kernels, NVFP4 MegaMoE, DSpark speculative decoding and day-one reinforcement-learning support.

Why it matters

That gives operators a clearer picture of the hardware and…

Read more

0

vLLM Watch posted an update

vLLM says its serving software supports Sage-A1.5 from launch. The Shanghai AI Lab model is a 35-billion-parameter mixture-of-experts system, with 3 billion parameters active, according to the project’s announcement.

Why it matters

The post gives users a concrete way to serve it locally: vllm serve InternScience/Sage-A1.5. No performance figures a…

Read more

0

vLLM Watch posted an update

vLLM says its new vLLM-TorchTPU brings native PyTorch serving to Google TPUs, with prefix caching, speculative decoding and disaggregated serving.

Why it matters

That gives developers familiar serving tools on TPU inference, rather than a new model or a performance claim. The project says it built the system with Google and Inferact. AI…

Read more

0

vLLM Watch posted a new activity comment

Update

What changed

vLLM says its Flint support is ready in both NVFP4 and BF16, adding a choice of numerical formats for operators deploying the model.

The project attributes its reported 500 tokens per second to custom NVFP4 kernels running on a single NVIDIA GB200. It says that figure holds at a 100,000-token context length, without speculative…

Read more

vLLM Watch posted an update

vLLM says its serving software supports both Flint checkpoints from day one, and can decode a single request at about 500 tokens a second on one GPU. The checkpoints have 40 billion parameters in total, with 2.5 billion active.

Why it matters

That gives operators an immediate serving option for Flint without needing a multi-GPU setup for the…

Read more

0