audio.cpp adds Kokoro and OpenVoice voice-cloning recipe
AI summary
A developer added an any-tts recipe to audio.cpp that pairs Kokoro with OpenVoice ToneColorConverter, combining speech synthesis and voice conversion without retraining the TTS model. The developer reported about 600 MB of additional peak VRAM, a 10% CUDA slowdown and roughly half the original CPU speed, while still running at twice real time. They described the Kokoro results as "pretty good," but the quality assessment was not independently verified.
In-depth insight
The recipe provides a reported way to add target-speaker voice conversion and voice-cloning capabilities to existing TTS models without retraining them, while documenting concrete memory and performance overheads.
While looking for audio models, I came across OpenVoice. It's getting pretty old, but surprisingly still ranks high on TTS leaderboards.
What caught my attention was its design: essentially TTS + voice conversion. The ToneColorConverter can run independently of the TTS model.
That got me thinking: Can we pair it with other TTS models?
I tried it with Kokoro, and the results were pretty good!
So I added a small recipe any-tts to audio.cpp for Kokoro + OpenVoice ToneColorConverter. Recipes were introduced in v0.9.2 as predefined pipelines that combine multiple audio models.
The overheads:
You can also try pairing any tiny TTS model with ToneColorConverter. The TTS model determines the speech content, pronunciation, and prosody, while the VC model transfers the target speaker's voice characteristics. The built-in voice you choose can also affect the final output quality.
A simple way to give existing TTS models voice cloning capabilities without retraining them! submitted by /u/Acceptable-Cycle4645
[link] [comments] |