MiTA

Headlines · 1

1. 5d
   NeurIPS 2026: attention as an FFN — BUPT's MiTA rethinks efficient attention

   Mixture-of-Top-k Attention (MiTA), proposed by a Beijing University of Posts and Telecommunications team, has been accepted at NeurIPS 2026, with the paper (arXiv 2602.01219) and code released. The work expands standard attention into a fast-weights FFN whose weights are generated by the current input and whose hidden width grows with sequence length, and uses that framing to organise prior efficient-attention methods into compression and routing routes plus a five-dimension taxonomy. MiTA derives m landmark queries by average pooling to build m reusable Top-k routed experts plus one shared expert, reaching 79.8% Top-1 on ImageNet-1K with DeiT-S versus 79.9% for full attention; the authors report 58.91% average accuracy on LRA (58.19% for full attention) with total training time cut from 10.6 to 2.4 hours on a single 24GB RTX 4090, and roughly 19x, 56x and 159x inference throughput at 64K, 256K and 1024K sequence lengths.

   36氪 人工智能 · 🔥 0

Experience and discussion from the community

Share my MiTA experienceAsk about MiTA