Humans Edge Speech Models on Cocktail-Party Audio Benchmark
A project by cloud_kj tested whether identifying one target voice amid background speech could distinguish humans from speech models. Across 40 synthetic clips and 146 human transcriptions from 18 sessions, humans averaged 0.71 word accuracy, compared with 0.54 for instructed Gemini 3.8 Flash and 0.55 for Qwen3-ASR 0.6B; the corresponding pass rates were 77% and 55%. The author concluded that the human-model gap was too narrow for the setup to work reliably as an audio CAPTCHA.
The released code, data, and audio corpus provide a reproducible test of whether current speech models can handle voice separation in synthetic cocktail-party conditions.
Hello - I wanted to share a small project I worked on test whether picking one voice out of background chatter could separate humans from machines, in the spirit of an audio CAPTCHA.
Setup
- 40 synthetic clips: one target sentence read by a synthetic voice, mixed over multiple background voices reading random short fragments (staggered starts, 4 dB attenuation)
- 146 human transcriptions from 18 sessions, unlimited replays allowed
- 8 model configurations, one pass per clip: Whisper base/small, Vosk, Wav2Vec2, Qwen3-ASR 0.6B, Parakeet CTC 0.6B, and Gemini 3.8 Flash with a naive prompt and with an instruction to focus on the main speaker
- Primary metric: word accuracy (1 − WER, floored at 0); default passing threshold = word accuracy ≥ 0.5
TL;DR: humans are better at the task but the gap is too narrow to be a reliable CAPTCHA
Results
- Humans 0.71 mean word accuracy vs. 0.54 for instructed Gemini (and 0.55 for Qwen3-ASR)
- Pass rate: 77% humans vs. 55% for Gemini
- Note: no threshold from ≥0.5 to ≥0.9 gets near my initial rough mental target: humans > 80% and machines < 20%
- Paired same-clip gap vs. Gemini: +0.13 (95% CI +0.03 to +0.23, session-level bootstrap); AUC 0.62
- The instruction prompt alone lifted Gemini from 0.34 to 0.54
- Replays didn't measurably help humans; people replayed the clips that were hardest for the models too
- Misses rise toward the end of the sentence for humans and every model alike
Limitations: small sample (18 sessions, mostly native English speakers), a single synthetic setup, no source separation, and no large or commercial ASR beyond Gemini.
I dusted off my dormant blog and did a write-up (https://www.kelvinjiang.com/2026/10/can-machines-follow-voice-through-crowded-room.html) and also shared the code, data, and audio corpus: https://github.com/cloudkj/crosstalk
Feedback on the method is very welcome, especially ideas for audio that stays easy for people but is consistently hard for models!
submitted by /u/cloud_kj [link] [comments]
#Crosstalk#Gemini 3.8 Flash#Qwen3-ASR 0.6B#Whisper#Vosk#Wav2Vec2#Parakeet CTC 0.6B