Frontier audio models struggle to label ASMR sounds, developer benchmark finds
A developer training a text-to-ASMR model evaluated eight frontier audio models as sound labellers on 480 clips with the same prompt and 20 labels, reporting that all struggled with fine-grained trigger sounds. Gemini 3.1 Pro had the highest trigger F1 against human labels at 0.325, while Claude Opus 5.5 and Grok 4.7, given spectrogram images, scored 0.121 and 0.110; Gemini 3.1 Pro named brushing in 92% of clean FSD50K/ESC-50 clips but reached only 0.18 brushing F1 on real ASMR, and whispering F1 was 0.70 to 0.79. The author reports Gemini 3.1 Pro at $0.0088 per clip and MiMo at $0.0008, and the findings come from a single r/MachineLearning post without independent verification.
The report offers unusually concrete evidence that leading audio models struggle to label fine-grained foley sounds, and the author is asking volunteers to help tag the roughly 126k recordings needed to train text2asmr.
I'm training text2asmr, a text-to-ASMR model (whispering plus trigger sounds like tapping, crinkling, brushing). The generator needs every training clip tagged with the sounds in it, and there are about 126k recordings, so I tried frontier models as labellers. They were much worse than I expected, so here are the numbers.
Setup. Every judge got the same prompt and the same 20 labels on 480 clips:
- External: sound-effect clips from FSD50K and ESC-50 that humans had already labelled with one class, 40 per class. This measures recall.
- People: 6-second ASMR clips labelled on our site. A class counts as present when at least half of a clip's labellers ticked it. This measures F1.
| Judge | Trigger F1 vs people |
|---|---|
| Gemini 3.1 Pro | 0.325 |
| Gemini 2.5 Pro | 0.290 |
| GPT Audio 1.5 | 0.288 |
| GPT Audio | 0.264 |
| MiMo v2.6 Pro | 0.243 |
| Gemini 3.8 Flash | 0.212 |
| Claude Opus 5.5 (spectrogram image) | 0.121 |
| Grok 4.7 (spectrogram image) | 0.110 |
#Gemini 3.1 Pro#Gemini 2.5 Pro#Gemini 3.8 Flash#GPT Audio 1.5#GPT Audio#MiMo v2.6 Pro#Claude Opus 5.5#Grok 4.7#text2asmr#CLAP-ASMR