创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

audio.cpp 0.9.5新增本地实时数字人语音聊天原型

AI 总结

一名Reddit用户称,audio.cpp 0.9.5的WebUI新增语音聊天原型,由SoulX-FlashHead生成数字人、OWSM v4转录语音,并使用LFM3-Audio处理对话与语音合成。该管线会调用LFM3-Audio两次,视频生成阶段建议使用RTX 4090或更高配置;据报告,在RTX 5090上从提交问题到数字人说话约需850毫秒。作者还表示,实时性能需要cuDNN,缺少它会使推理速度减半并令显存占用翻倍。

深度见解

该原型将语音识别、对话音频、文本转语音和实时数字人生成组合成具体的本地多模态管线,但性能与硬件数据来自社区报告,尚未经过独立验证。

In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI. The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS. LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice. One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs. cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles. ~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency. submitted by /u/Acceptable-Cycle4645 [link] [comments]

#audio.cpp#SoulX-FlashHead#OWSM v4#LFM3-Audio#NVIDIA#cuDNN

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。