Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

sglang-omni

sglang-omni: A multi-stage serving runtime for omni, speech, and TTS models.

What problem it solves

Interactive multimodal and speech systems often require generation to span heterogeneous stages with different compute patterns, dependency structures, and resource needs. Serving these pipelines requires coordinated stage lifecycles, inter-stage transport, model integrations, and a unified API surface.

Key features

  • Coordinated multi-stage runtime — Models generation as coordinated preprocessing, encoding, autoregressive, talking, decoding, vocoding, and aggregation stages.

    Source: README, sglang_omni

  • Stage-specialized scheduling — Each stage uses a scheduler matched to its workload, including autoregressive scheduling and lightweight preprocessing or streaming vocoder loops.

    Source: README

  • Transport-aware execution — A control plane coordinates requests while a relay data plane transfers tensor payloads through supported backends.

    Source: README, sglang_omni_router

  • OpenAI-compatible API surface — Serving endpoints cover multimodal chat, speech generation, batch and streaming speech, uploaded voices, and transcription.

    Source: README

  • Multiple hardware backends — The runtime supports NVIDIA CUDA and Intel GPU configurations, with an experimental Apple Silicon path for Qwen3-ASR.

    Source: README, pyproject_xpu.toml

Who it may suit

This project may suit inference-system developers, model integrators, and operators building interactive multimodal, speech, or TTS serving pipelines.

Topics to discuss

  • Which stage combinations should receive new model-family integrations?

  • How should transport backends be selected for different deployment topologies?

  • Which production workloads most need additional scheduling and cache optimizations?


Written by ReadmeX AI from the project's README, source and public GitHub data; it may contain mistakes.

Read the original README https://github.com/sgl-project/sglang-omni#readme

ReadmeX opened this community for a trending project; it is not run by the project. Maintainers can claim it from the community page.

github.comhttps://github.com/sgl-project/sglang-omni

Comments

…