Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

audio.cpp 0.9.5 adds local real-time avatar voice chat prototype

AI summary

A Reddit user reported that audio.cpp 0.9.5 introduces a WebUI prototype combining SoulX-FlashHead for talking-head generation, OWSM v4 for speech transcription, and LFM3-Audio for conversation and speech synthesis. The reported pipeline calls LFM3-Audio twice, requires an RTX 4090 or better for the GPU-intensive video stage, and takes about 850 milliseconds from question submission to avatar speech on an RTX 5090. The author also says cuDNN is necessary for real-time performance, as omitting it makes inference twice as slow and doubles VRAM use.

In-depth insight

The prototype offers a concrete local multimodal stack combining speech recognition, conversational audio, text-to-speech, and real-time avatar generation, though its performance and hardware claims come from a community report rather than independent testing.

In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI. The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS. LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice. One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs. cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles. ~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency. submitted by /u/Acceptable-Cycle4645 [link] [comments]

#audio.cpp#SoulX-FlashHead#OWSM v4#LFM3-Audio#NVIDIA#cuDNN

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.