Study compares hybrid attention designs for long-context language models
机器之心 reports that a Fudan University and Shanghai Innovation Institute team led by Qiu Xipeng compared RoPE/NoPE full attention, sliding-window attention, and gated linear attention hybrids on long-context tasks. Across models from 376M to 3B parameters, the study found trade-offs between length extrapolation and continued long-context training, and proposed sliding-window linear attention to improve extrapolation. The researchers also analyzed how position-biased and position-free attention modules cooperate.
The work provides a systematic account of how different long-context attention components complement one another, along with concrete training and inference strategies for improving extrapolation.
机器之心 2026-11-01 12:00 四川
混合架构为什么有效，以及怎样设计得更有效？
本文第一作者柳潇然，复旦大学、上海创智学院 2022 级硕博连读生，导师邱锡鹏、何子薇。研究方向为长上下文大语言模型，曾在 ICLR、ICML、AAAI、EMNLP 等国际顶级人工智能会议发表论文十余篇。
背景介绍
#Fudan University#Shanghai Innovation Institute#OpenMOSS#RoPE#NoPE#Sliding-Window Attention#Gated Linear Attention#Gated DeltaNet#Sliding-Window Linear Attention