SGLang adds Low-Latency Mode with plugin kernels for NVIDIA and AMD GPUs
The SGLang, NVIDIA and AMD teams introduced SGLang Low-Latency Mode, which brings specialized low-latency kernels into the SGLang serving stack through a plugin framework, with NVIDIA kernels in sgl-nvidia-mega and AMD kernels in sgl-amd-mono. In AgentX trace replay with DeepSeek-V4.1-Flash with DSpark, the teams report P90 per-user token generation rising from 671.2 to 928.4 tokens/s (1.38x) on four NVIDIA GB300 GPUs at one concurrent session, and from 331.6 to 516.0 tokens/s (1.56x) on four AMD MI355X GPUs at four concurrent sessions, each against the SGLang baseline on the same hardware. The NVIDIA plugin also improved per-user decode speed by 12-38% at 1-8 concurrent requests on an 8K-input / 1K-output workload, and the teams say megakernel work is underway for GLM-5.3, K3 and MiniMax M3.
The plugin framework lets kernel teams ship model- and hardware-specific optimizations without adding every specialized implementation to SGLang's core codebase.
SGLang Low-Latency Mode: Kernels and Runtime
SGLang, NVIDIA and AMD teamsOctober 18, 2026
Low-latency inference offers many opportunities for optimizations tailored to specific models, hardware and workloads. These optimizations often require extensive kernel fusion and duplicate code across implementations, making them difficult to maintain. To support this specialization, we introduce SGLang Low-Latency Mode. It brings specialized kernels into SGLang through a plugin framework and delivers substantial latency reductions on both NVIDIA and AMD GPUs. In AgentX trace replay, these optimizations improve P90 per-user token generation speed by up to 1.38× on NVIDIA GB300 and 1.56× on AMD MI355X compared with the respective SGLang baselines.
AgentX · DeepSeek-V4.1-Flash with DSpark · NVIDIA GB300 and AMD MI355X.
Table of Contents
#SGLang#NVIDIA#AMD#DeepSeek-V4.1-Flash#DSpark#GLM-5.3#MiniMax M3