英伟达发布长上下文稀疏注意力指南
NVIDIA Technical Blog 发布面向长上下文推理的实用指南,重点介绍训练进模型的稀疏注意力,并区别于 BLASST、MInference 和 Quest 等训练后方法。文章利用生产级 FP8 内核测试索引器参数、Top-k 扩展性、推测解码及词元级与块级选择,并提出八项面向 NVIDIA GPU 的设计建议。NVIDIA 称,在 128K token 长度下,密集注意力可能占用一层中大部分预填充时间,而稀疏注意力的主要剩余成本转移到了索引器。
这份面向硬件的详细指南为降低长上下文模型的注意力计算和内存流量提供了具体设计选择。
Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths, attention becomes the most expensive part of a transformer layer. Dense attention scores every query against every key, so its cost per query grows with the context length N, and at 128K tokens it can take most of the prefill time in a layer.
The model-level solution is sparse attention, a transformer technique in which each query attends to only a small subset of the context, chosen per query, instead of all of it. This cuts both the compute and the memory traffic of attention on long context.
This post is a practical primer for model developers designing sparse attention that runs well on NVIDIA GPUs. It focuses on model-level sparse attention, where sparsity is trained into the model, rather than drop-in methods applied after training such as BLASST, MInference, or Quest. A production FP8 kernel is used to sweep the indexer’s design knobs (query heads, key heads, head dimension, compression ratio, and speculative-decoding depth), measure how Top-k cost scales, and compare token-level with block-level core attention.
#NVIDIA#DeepSeek V3.2#DeepSeek-R1#GLM 5.2#GLM 5.3 Flash#LongCat LSA-Flash