SGLang launches beta plugin for AWS Trainium inference and RL
The SGLang Team, in collaboration with AWS, announced a beta plugin for running LLM inference serving and reinforcement learning rollouts on AWS Trainium2 and Trainium3. Built on TorchNeuron, the beta initially covers Qwen3, Qwen3-MoE-30B-A3B, DeepSeek-V2-Lite, GLM-4, and GLM-5.2/5.3, with parallelism, Radix Cache, BF16, FP8, and an asynchronous scheduler. In an early end-to-end test, SGLang generated Qwen3-MoE-30B-A3B rollouts on Trainium3 while Miles trained a GRPO policy with Megatron on one H200 node; performance work remains ongoing.
The plugin adds an AWS Trainium path for both open-model serving and heterogeneous RL pipelines, allowing rollout generation and policy training to run on different hardware.
On this page
Over the past year, we have continued to expand the hardware SGLang supports, alongside launch-day support for new open models and RL rollouts in Miles. Now, we are collaborating with AWS to bring SGLang to AWS Trainium.
As frontier AI moves toward heterogeneous hardware, AWS Trainium represents a unique opportunity. Purpose-built by AWS for AI and available at scale in the cloud, Trainium is designed to deliver strong price-performance for high-volume workloads. Open-model serving and RL rollout generation are prime examples: both run continuously, and cost per token directly determines how much capacity teams can deploy. Our goal is to make SGLang a first-class option for running these workloads on Trainium.
In close collaboration with AWS, and in conjunction with the TorchNeuron launch, we are announcing the beta of the SGLang Trainium plugin, built on the newly released TorchNeuron. The beta enables both inference serving and RL rollouts on Trainium.
#SGLang#AWS#SGLang Trainium Plugin#TorchNeuron#AWS Trainium2#AWS Trainium3#Qwen3#Qwen3-MoE-30B-A3B#DeepSeek-V2-Lite#GLM-4#GLM-5.2/5.3#Miles