Deepgram says AWS Trainium handled up to 70% more audio per dollar

Deepgram says it processed 11–70% more audio per dollar on AWS Trainium2 than on the compute options it had been using. The work shows how model and software optimisation can change the economics of AI inference, not just which chip runs it.

AWS AI Watch analysis

What happened

In a post published on 19 October, AWS describes how it and Deepgram ported Deepgram’s Nova-3 speech-to-text model to Trainium2 for offline batch workloads such as contact-centre analytics and medical transcription. The reported comparison uses instance pricing at the time of evaluation; it is not a claim that Trainium is cheaper for every workload.

The team used TorchNeuron to run PyTorch on Trainium, tuned model kernels and built a Rust serving runtime. AWS says the final setup ran eight model replicas on one EC2 trn2.3xlarge instance. Deepgram’s offline, throughput-focused tests swept batch sizes from 1 to 128.

Why it matters

The headline result is a range, not a universal saving: 11–70% more audio per dollar than Deepgram’s previous compute options, under its batch workload and evaluation conditions. That makes this a useful example for operators weighing the full cost of inference, where model changes, runtime overhead and hardware all matter.

The post also reports individual engineering gains: a hand-tuned feed-forward kernel had half the latency of its reference kernel at batch size 96; a rewritten depthwise convolution ran 1.5 times faster there. A Rust runtime improved throughput by 7.5% over a multiprocessing Python baseline at the throughput-maximising batch size. Those figures describe different parts of the system and should not be added together like a particularly optimistic shopping receipt.

Our read

The strongest takeaway is not that one chip has won, but that inference performance depends on the work around the chip as well as the chip itself. Deepgram and AWS report a meaningful result for a specific, high-volume batch workload. Teams serving interactive speech or different models should look for results on their own workloads before treating the cost range as a forecast.

What to watch

• Whether Deepgram deploys the optimised setup beyond the workloads described.
• How the reported price-performance compares with alternatives on the same workload and evaluation basis.
• Whether the Trainium port and software tools become easier for other model teams to use.
• How results change for real-time inference, which the post does not benchmark.

Discussion spark: For teams choosing inference hardware, should workload-specific price-per-output results carry more weight than general-purpose benchmarks?

not affiliated with or endorsed by Amazon Web Services (AWS)