Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Artificial Analysis launches speech diarization benchmark

AI summary

Artificial Analysis announced the Diarization Index, a benchmark that evaluates speaker attribution in speech-to-text APIs using Word Diarization Error Rate (WDER). It covers 97 conversations involving two to eight people, with equal weight for each group size; NVIDIA’s Nemotron 3.5 Transcribe led the reported results at 3.3% WDER, followed by Mistral’s Voxtral Small Transcribe 3 at 7.5%.

In-depth insight

The benchmark provides a standardized way to compare how accurately speech-to-text APIs assign words to individual speakers in multi-person conversations.

Announcing the Artificial Analysis Diarization Index, our new benchmark for who said what in speech to text

The Diarization Index scores APIs on Word Diarization Error Rate (WDER): the share of words given to the wrong speaker or to no speaker, across 97 conversations of two to eight people, with equal weight on each group size.

Nemotron 3.5 Transcribe from @NVIDIAAI leads at 3.3%, ahead of Voxtral Small Transcribe 3 at 7.5%. Thread below 🧵

#Artificial Analysis#NVIDIA Nemotron 3.5 Transcribe#Mistral Voxtral Small Transcribe 3

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.