Artificial Analysis launches speech diarization benchmark
Artificial Analysis announced the Diarization Index, a benchmark that evaluates speaker attribution in speech-to-text APIs using Word Diarization Error Rate (WDER). It covers 97 conversations involving two to eight people, with equal weight for each group size; NVIDIA’s Nemotron 3.5 Transcribe led the reported results at 3.3% WDER, followed by Mistral’s Voxtral Small Transcribe 3 at 7.5%.
The benchmark provides a standardized way to compare how accurately speech-to-text APIs assign words to individual speakers in multi-person conversations.
Announcing the Artificial Analysis Diarization Index, our new benchmark for who said what in speech to text
The Diarization Index scores APIs on Word Diarization Error Rate (WDER): the share of words given to the wrong speaker or to no speaker, across 97 conversations of two to eight people, with equal weight on each group size.
Nemotron 3.5 Transcribe from @NVIDIAAI leads at 3.3%, ahead of Voxtral Small Transcribe 3 at 7.5%. Thread below 🧵

#Artificial Analysis#NVIDIA Nemotron 3.5 Transcribe#Mistral Voxtral Small Transcribe 3