Baseten says its Qwen3 speech-to-text stack transcribes audio up to seven times faster

Baseten says it has optimised Qwen3-ASR 1.7B for batch transcription, achieving up to seven times the throughput of the model running on unmodified vLLM. On a single RTX PRO 6000, its tests also handled hour-long recordings in one request and put the estimated cost at about $0.0013 per audio hour.

Baseten Watch analysis

What happened

The changes combine native audio decoding, an FP8 key-value cache and fewer CPU-to-GPU synchronisation delays. Baseten says its optimised setup reached 3,128 times real-time speed on hour-long files, compared with a plateau of about 380 times real-time on vanilla vLLM. The unmodified setup could not fit audio longer than ten minutes into one request.

For five-minute clips at 64 concurrent requests, Baseten reports median latency falling from 50 seconds to 7.3 seconds. Its cost estimate uses the company’s list price for a dedicated RTX PRO 6000 and assumes the GPU is saturated; it puts the same model on vanilla vLLM at $0.011 per audio hour.

Why it matters

Batch transcription is a different job from turning speech into text quickly enough for a live voice agent. For archives of calls, meetings or podcasts, more audio processed per GPU can mean lower costs and less waiting. Fitting an hour-long file into a single request also removes a practical limit that Baseten says affected the unmodified setup.

These are Baseten’s benchmark results, not a universal promise about every deployment. They do, however, show how much the serving stack can matter alongside the model itself. The model may be open-weight; getting useful speed from it still takes engineering.

Our read

The standout result is not a magical new model, but the less glamorous work of decoding audio efficiently and keeping the GPU busy. If you run large batch jobs, the reported gains make this stack worth a look. Check the hardware, concurrency and pricing assumptions before treating the headline cost as your own forecast.

What to watch

• Whether independent testing reproduces the throughput and latency results.
• How the results change on other GPUs, workloads and levels of concurrency.
• Whether Baseten’s deployment remains cost-effective at real-world utilisation.

Discussion spark: For speech-to-text workloads, should teams prioritise an open-weight model they can optimise themselves, or a managed service with simpler deployment even if it costs more?

not affiliated with or endorsed by Baseten