Triton

Headlines · 2

1. 4d
   vLLM: Kimi K3 on AMD MI355X tops NVIDIA HGX B300 throughput per GPU

   A vLLM blog post details the engineering behind serving Kimi K3 on an eight-GPU AMD Instinct MI355X system with vLLM, covering cache reuse of conversation state, eliminating duplicate MoE work across GPUs, fusing memory traffic, an AMD AITER assembly path that speeds long-context MLA decoding, and parallelized KDA prefill and speculative decoding. Citing a SemiAnalysis InferenceX snapshot from October 9, 2026, it reports about 7,567 total tokens/s/GPU at 50 tokens/s/user P90 interactivity versus 6,083 for NVIDIA HGX B300, a 24.4% advantage. Using modeled TCO of $1.50 versus $2.26 per GPU-hour, it claims 87.4% more tokens per dollar; the figures come from a vendor-adjacent source and modeled cost assumptions, with no independent replication reported.

   vLLM Blog · 🔥 0
2. 6d
   AWS details fused MoE kernel optimizations on SageMaker, up to 10% throughput gain

   An AWS Machine Learning Blog post describes several fused Mixture-of-Experts (MoE) kernel optimizations that AWS developed with the vLLM community and open sourced. A Swap-AB transposition that moves the large expert dimension into the leading WGMMA operand slot delivered a 2–4% end-to-end output throughput improvement on H200 and was upstreamed to vLLM 0.24.0 and later. Routing FP8 tensor-quantized MoE workloads to the Triton backend instead of FlashInfer+CUTLASS added about 10% throughput (available in the AWS LMI v28 container), while adding do_not_specialize annotations to variable-length parameters removed repeated Triton JIT recompilations and improved time to first token by 9% on H200 at 8-way tensor parallelism (vLLM 0.22.0 and later). AWS says the optimizations together yield up to 10% higher throughput and up to 9% faster TTFT.

   AWS Machine Learning Blog · 🔥 0

Experience and discussion from the community

Share my Triton experienceAsk about Triton