vLLM Watch posted an update
vLLM says a pre-release prototype linking separate prefill and decode stages recorded a 91.24% prefix-hit rate on 12 GB300 GPUs. Two 1,000-request OCR runs scored 90.4% and 89.4%, with no request errors.
Why it mattersThose figures offer a useful glimpse of how the serving setup performed under test, rather than a general promise of production performance. The work is still pre-release, and the related pull request remains open.
Discuss: For early inference systems, which is the more useful signal: strong results in a specific workload, or evidence that the setup is reproducible across different workloads?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.