WorldSonus：为世界模型带来实时空间音效

Hugging Face · Papers10-05 20:00Pengjun Fang, Jingyi Fa, Kam Man Wu, Jiaming Wang, Haoyuan Huang, Yaguang Wu, Xiangjun Huang, Ziyang Ma, Weijia Chen, Hongyu Liu, Zeyue Tian, Qifeng Chen

群聊原文

AI 总结

arXiv 论文提出 WorldSonus，一个面向世界模型的交互式视频转音频框架，目标是让生成的虚拟环境配上同步声音。它采用流式因果自回归扩散架构，论文称实时因子（RTF）为 0.41，并通过分块索引的提示调度支持生成过程中的声音指令控制。作者还使用立体声与 ambisonic 数据提供空间监督，使输出立体声与场景几何和相机运动对齐。

为什么重要：世界模型目前大多只有画面没有声音，实时且空间一致的声音合成是让生成环境可用于交互体验的关键一步。

WorldSonus

来源原文Hugging Face · Papers · 约 3 分钟读完

Computer Science > Sound

arXiv:2610.08760 (cs)

[Submitted on 6 Oct 2026]

Title:WorldSonus: Bringing Sound to Worlds

Authors:Pengjun Fang, Jingyi Fa, Kam Man Wu, Jiaming Wang, Haoyuan Huang, Yaguang Wu, Xiangjun Huang, Ziyang Ma, Weijia Chen, Hongyu Liu, Zeyue Tian, Qifeng Chen

View a PDF of the paper titled WorldSonus: Bringing Sound to Worlds, by Pengjun Fang and 11 other authors

View PDF HTML (experimental)

Abstract:Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: this https URL

Comments:	25 pages, 4 figures, 16 tables. Project page: this https URL
Subjects:	Sound (cs.SD); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Audio and Speech Processing (eess.AS)
Cite as:	arXiv:2610.08760 [cs.SD]
	(or arXiv:2610.08760v1 [cs.SD] for this version)
	https://doi.org/10.48550/arXiv.2610.08760 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Pengjun Fang [view email]
[v1] Tue, 6 Oct 2026 17:50:18 UTC (10,428 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled WorldSonus: Bringing Sound to Worlds, by Pengjun Fang and 11 other authors

View PDF
HTML (experimental)
TeX Source

license icon view license

Additional Features

Audio Summary

Current browse context:

cs.SD

< prev | next >

new | recent | 2026-10

Change to browse by:

cs
cs.AI
cs.CV
eess
eess.AS

References & Citations

BibTeX formatted citation

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

阅读原始来源 →

登录 ReadmeX

资讯