Deepgram ports Nova-3 speech model to AWS Trainium2, reports 11–70% more audio per dollar
Deepgram worked with AWS's Forward Deployed Engineering team to port its Nova-3 speech-to-text model from standard PyTorch code to AWS Trainium2, tuning it for offline batch workloads such as contact-center analytics and medical transcription. The final setup runs eight model replicas on a single Amazon EC2 trn2.3xlarge instance, using TorchNeuron, prebuilt NKILib kernels, hand-written NKI kernels and a Rust data-parallel runtime. According to the AWS blog, that configuration processed 11–70 percent more audio per dollar than the compute options Deepgram had been running on those offline throughput-oriented workloads, measured with instance pricing at evaluation time; no independent verification is cited.
For high-volume transcription, inference efficiency determines how much audio a fixed infrastructure budget can serve, so accelerator portability and kernel-level tuning bear directly on unit costs; the reported gains are vendor-measured and pricing-dependent.
As voice becomes a first-class modality for AI, Deepgram’s models span speech-to-text, text-to-speech, and voice agents. They now process quadrillions of audio tokens a year for more than 200,000 developers and 1,400 organizations. Customers use them through the Deepgram API or run them in their own environments, including on Amazon SageMaker AI.
For Deepgram, serving efficiency determines how much audio the system can process for a given infrastructure budget. High-volume batch workloads put heavy pressure on that budget, so the more efficiently each accelerator hour is used, the more audio Deepgram can serve on the same fleet. That made inference efficiency a natural target and Deepgram turned to AWS AI chips to improve it. They worked with AWS to bring its Nova-3 speech-to-text model to AWS Trainium2 for offline batch workloads such as contact-center analytics and medical transcription.
The engagement began with Deepgram delivering Nova-3 as standard PyTorch code. Over several months, Deepgram and the AWS Forward Deployed Engineering (FDE) team ported Nova-3 to AWS Trainium2 in PyTorch reconstruction and tuned it for the new hardware. The final configuration runs eight model replicas on a single Amazon Elastic Compute Cloud (Amazon EC2) trn2.3xlarge instance. On Deepgram’s offline, throughput-oriented batch workloads, that setup processed 11–70 percent more audio per dollar than the compute options Deepgram had been running. We measured this using instance pricing at the time of the evaluation. These gains help Deepgram scale efficiently, keep shipping better models, and bring them to customers at lower prices. These benefits can extend to customers who self-host Deepgram models in their own virtual private cloud (VPC) on Amazon SageMaker AI. There, Deepgram and AWS continue to improve ease of setup, reliability, and observability.
#Deepgram#Deepgram Nova-3#AWS Trainium2#Amazon EC2 trn2.3xlarge#AWS Neuron#TorchNeuron#Neuron Kernel Interface (NKI)#Amazon SageMaker AI#AWS Forward Deployed Engineering