PyTorch adds FlyDSL backend support for AMD MI350 GPUs
PyTorch Blog reports that AMD Instinct MI350-series GPU users can access FlyDSL kernels through existing PyTorch operator APIs. The backend covers dense and grouped GEMM, MXFP8 and MXFP4 scaled GEMM, attention, RMSNorm and TopK. GEMMs use the optional backend through torch.compile, eligible RMSNorm and TopK calls can dispatch automatically in eager mode, and unsupported inputs retain existing PyTorch implementations. FlyDSL attention kernels are included in AOTriton 0.14.2b or later. Official tests on MI355X reported a 1.10x geometric-mean speedup for the BF16 dense-GEMM suite, while end-to-end vLLM A/B tests reached whole-request speedups of 1.13x for BF16, 2.04x for MXFP8 and 3.01x for MXFP4. These results depend on the workload, model and concurrency, and were not independently verified in the supplied reports.
The integration gives PyTorch users access to AMD-specific FlyDSL kernels through familiar APIs and dispatch paths. The reported model-level gains were strongest in the low-precision MXFP8 and MXFP4 vLLM tests, although their applicability depends on the workload.
PyTorch users on AMD MI350-series GPUs can now use FlyDSL through existing operator APIs for dense and grouped GEMM, fused multi-head attention (FMHA/Attention), MXFP8/MXFP4 scaled GEMM, RMSNorm, and TopK, without requiring applications to call custom kernels. The optional backend supports GEMMs through torch.compile, while eligible RMSNorm and TopK calls dispatch automatically in eager mode when it is available and enabled. Unsupported inputs retain existing PyTorch implementations. FlyDSL attention kernels ship as an additional backend in AOTriton, the library behind PyTorch’s ROCm F.scaled_dot_product_attention (SDPA), starting with the 0.14b series.
Across the reported kernel-level operator suites, FlyDSL provides a 1.10x geometric-mean speedup for the 15-shape BF16 dense-GEMM suite over the faster of ATen and Triton at each shape, 1.58x and 1.68x geometric-mean speedups for MXFP8 and MXFP4, respectively, over ATen across 17 shapes per format, 1.20x geometric mean over Triton across all 24 grouped-GEMM cases, and 1.17x–3.66x over ATen across the 22 RMSNorm cases. For the 10 small-K TopK cases, the geometric mean is 4.82x over ATen with deterministic algorithms disabled. These gains are most relevant when the supported operations account for a significant share of model runtime. For attention, AOTriton 0.14.2b’s published gfx950 throughput ranges from 0.92x to 2.34x of 0.13b for forward and 0.93x to 2.47x for backward, across head dimensions and causal modes. In end-to-end vLLM A/B tests, whole-request speedup reaches 1.13x for BF16, 2.04x for MXFP8, and 3.01x for MXFP4, with the result depending on model and concurrency.
#PyTorch#AMD#FlyDSL#AMD Instinct MI350#AOTriton#ROCm#ATen#Triton#TorchInductor#vLLM