FlashAttention

Headlines · 3

1. Nov 3
   Roofline for LLM Inference: Performance Budgets from Compute and Bandwidth

   A personal-blog article, presented as the first installment of an eight-part “Inference INFRA Methodology” series, applies the Roofline model to GPU inference. It estimates latency with T ≥ max(W/F, Q/BW) and effective performance with P ≤ min(F, I×BW), using arithmetic intensity, the hardware ridge point and actual HBM traffic to diagnose GEMM and attention bottlenecks. H20 examples and the article’s own measurements distinguish when quantization, batching and operator fusion may help. A series schedule covers communication, distributed inference, hardware, attention, MoE, PD/EPD and core optimizations. The benchmarks are self-reported and have not been independently verified here.

   掘金 · 人工智能 · 🔥 0
2. Nov 2
   LlamaIndex reports nearly 10x faster token generation for OCRia-1 on one B200

   LlamaIndex reports that its optimizations raised OCRia-1 token generation from about 285 to 2,800 tokens per second on a single NVIDIA B200 GPU, a vendor-reported result not independently verified here. The stack combines workload-aware mixed-precision quantization, DFlash2 speculative decoding, ReplaySSM, custom kernels and 17 additional optimizations. LlamaIndex says OCRia-1 is available through LlamaParse across all tiers and plans to contribute generally applicable improvements to vLLM.

   LlamaIndex Blog · 🔥 0
3. Oct 12
   NeurIPS 2026: attention as an FFN — BUPT's MiTA rethinks efficient attention

   Mixture-of-Top-k Attention (MiTA), proposed by a Beijing University of Posts and Telecommunications team, has been accepted at NeurIPS 2026, with the paper (arXiv 2602.01219) and code released. The work expands standard attention into a fast-weights FFN whose weights are generated by the current input and whose hidden width grows with sequence length, and uses that framing to organise prior efficient-attention methods into compression and routing routes plus a five-dimension taxonomy. MiTA derives m landmark queries by average pooling to build m reusable Top-k routed experts plus one shared expert, reaching 79.8% Top-1 on ImageNet-1K with DeiT-S versus 79.9% for full attention; the authors report 58.91% average accuracy on LRA (58.19% for full attention) with total training time cut from 10.6 to 2.4 hours on a single 24GB RTX 4090, and roughly 19x, 56x and 159x inference throughput at 64K, 256K and 1024K sequence lengths.

   36氪 人工智能 · 🔥 0

Experience and discussion from the community

Share my FlashAttention experienceAsk about FlashAttention