Reddit user benchmarks MOSS-Transcribe-Diarize 0.9B on a MacBook
A Reddit user in r/LocalLLaMA ran MOSS-Transcribe-Diarize 0.9B on a 17:29 four-speaker AMI test meeting using the model's README transformers code in fp32 with sdpa attention, greedy decoding, no hotwords and no VAD, on an M5 Pro MacBook Pro with 64 GB via MPS. The user reports the run took 714 seconds for the 1,049-second file, about 0.68x real time, and found all four speakers; with a 0.25-second collar and best speaker mapping, speaker confusion was 0.08% and overall DER was 15.4%. The author notes this is one meeting scored on who and when, and asks for pyannote 3.1 plus Whisper comparisons on Apple silicon.
It offers a concrete, if single-meeting, data point on running a combined transcription-and-diarization model locally on Apple silicon.
Wanted to see how one model that does transcription, speaker labels and timestamps in a single pass handles a real meeting, so I ran MOSS-Transcribe-Diarize 0.9B on a four person meeting from the AMI test set and scored its speaker turns against AMI's reference.
Setup: 17:29 of AMI headset mix audio with four speakers (diarizers-community/ami, test split, row 12; AMI is CC BY 4.0). Their README's transformers code, fp32, sdpa attention, greedy decoding, no hotwords, no VAD, nothing else in the pipeline. M5 Pro MacBook Pro with 64 GB, running on the GPU through mps (on a Mac, set the device to "mps" yourself).
It took 714 s for the 1,049 s file, about 0.68x real time, and it found all four speakers. The labels are the standout: with the usual 0.25 s collar and the best speaker mapping, speaker confusion was 0.08%, so when it writes a line, it's attributed to the right person almost every time. Overall DER at that collar was 15.4%.
It also handles people talking over each other. 87 of its 317 segments start before the previous one ends, and every one of those is a different speaker, so interruptions come out as overlapping lines with both people labeled.
First few lines of raw output, untouched:
[10.88][S01]Are we we're not allowed to dim the lights so people can see that a bit better.[14.56] [17.86][S02]Yeah.[18.21] [18.84][S01]Okay, that's fine.[20.28] [22.31][S01]Am I supposed to be standing up there?[23.76] [25.08][S03]So we've got both of these clipped on.[26.56] [29.04][S03]Is she gonna answer me or not?[30.24] [29.84][S01]Yeah, I've got.[30.44] This is one meeting, scored on who and when, since that's what the AMI reference I used covers.
If anyone has pyannote 3.1 plus Whisper numbers on the AMI test meetings on Apple silicon, I'd like to see DER and runtime on the same files.
#MOSS-Transcribe-Diarize 0.9B#Apple#MacBook Pro#AMI#pyannote#Whisper