sglang-omni: A multi-stage serving runtime for omni, speech, and TTS models.
What problem it solves
Interactive multimodal and speech systems often require generation to span heterogeneous stages with different compute patterns, dependency structures, and resource needs. Serving these pipelines requires coordinated stage lifecycles, inter-stage transport, model integrations, and a unified API surface.
Key features
Coordinated multi-stage runtime — Models generation as coordinated preprocessing, encoding, autoregressive, talking, decoding, vocoding, and aggregation stages.
Source: README, sglang_omni
Stage-specialized scheduling — Each stage uses a scheduler matched to its workload, including autoregressive scheduling and lightweight preprocessing or streaming vocoder loops.
Source: README
Transport-aware execution — A control plane coordinates requests while a relay data plane transfers tensor payloads through supported backends.
Source: README, sglang_omni_router
OpenAI-compatible API surface — Serving endpoints cover multimodal chat, speech generation, batch and streaming speech, uploaded voices, and transcription.
Source: README
Multiple hardware backends — The runtime supports NVIDIA CUDA and Intel GPU configurations, with an experimental Apple Silicon path for Qwen3-ASR.
Source: README, pyproject_xpu.toml
Who it may suit
This project may suit inference-system developers, model integrators, and operators building interactive multimodal, speech, or TTS serving pipelines.
Topics to discuss
Which stage combinations should receive new model-family integrations?
How should transport backends be selected for different deployment topologies?
Which production workloads most need additional scheduling and cache optimizations?
Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.
Read the original README https://github.com/sgl-project/sglang-omni#readme
ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.
Comments
…