NVIDIA Hopper

Headlines · 1

1. 6d
   AWS details fused MoE kernel optimizations on SageMaker, up to 10% throughput gain

   An AWS Machine Learning Blog post describes several fused Mixture-of-Experts (MoE) kernel optimizations that AWS developed with the vLLM community and open sourced. A Swap-AB transposition that moves the large expert dimension into the leading WGMMA operand slot delivered a 2–4% end-to-end output throughput improvement on H200 and was upstreamed to vLLM 0.24.0 and later. Routing FP8 tensor-quantized MoE workloads to the Triton backend instead of FlashInfer+CUTLASS added about 10% throughput (available in the AWS LMI v28 container), while adding do_not_specialize annotations to variable-length parameters removed repeated Triton JIT recompilations and improved time to first token by 9% on H200 at 8-way tensor parallelism (vLLM 0.22.0 and later). AWS says the optimizations together yield up to 10% higher throughput and up to 9% faster TTFT.

   AWS Machine Learning Blog · 🔥 0

Experience and discussion from the community

Share my NVIDIA Hopper experienceAsk about NVIDIA Hopper