FlashMask-3优化掩码分布式注意力
AI 总结
FlashMask-3论文面向长上下文模型训练提出掩码感知的分布式注意力，将基于FlashAttention-3的Hopper内核、零通信掩码定位和计算通信重叠结合起来。作者报告其单卡TFLOPs、分布式吞吐和端到端训练吞吐均高于所列基线，但这些结果尚未经独立验证。
深度见解
该工作聚焦长上下文训练中的具体瓶颈，即在多GPU上处理结构化注意力掩码，同时维持计算与通信重叠。
| Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node context parallelism, making the mask a dominant bottleneck. FlashMask's column-wise sparse mask compresses them from O(n2) to O(n) via four query-axis boundary vectors, covering broad structured patterns. Implemented for Ampere GPUs, it neither exploits FlashAttention-3's TMA/WGMMA warp-specialized pipeline on Hopper nor supports distribution. The column-wise sparse mask is indexed by key positions but stores query-axis interval boundaries. Under any multi-chunk context-parallel assignment, it reduces per-rank mask localization to a provably zero-communication \emph{clip--shift--max} on the boundary vectors, compatible with the single-GPU kernel. FlashMask-3 introduces a mask-aware Hopper kernel with three-role warp specialization atop FlashAttention-3's TMA/WGMMA pipeline, plus a mask-aware distributed extension: zero-communication mask localization, a sparsity-aware load balancer (HCC-IPO), and a mask-aware compute-communication overlap with mask-driven communication pruning, whose forward all-gather uses topology-aware hierarchical routing. On a single H100 across 12 masks at 4K--128K context, it improves achieved TFLOPs over FlashMask, FlexAttention, and MagiAttention by 45--141%, 34--71%, and 2--44%. At context-parallel degrees 4--32 with 8K sequence per device, it attains 31--75% compute-communication overlap efficiency and improves throughput over Megatron-Core CP ring, Megatron-Core CP all-gather, and MagiAttention by 1.80×--3.71×, 1.71×--2.92×, and 1.09×--1.46×. In end-to-end training of a 118B-A14B MoE at 128K on 128 H100 GPUs, it raises throughput by 7.3--11.0% over a Dual Chunk all-gather baseline, with attention forward+backward 24.5--44.9% faster. submitted by /u/pmttyji [link] [comments] |
#FlashMask-3#FlashMask#FlashAttention-3#FlexAttention#MagiAttention#Megatron-Core#Dual Chunk#Nvidia H100#PaddlePaddle