Sliding-Window Linear Attention
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Study compares hybrid attention designs for long-context language models
机器之心 reports that a Fudan University and Shanghai Innovation Institute team led by Qiu Xipeng compared RoPE/NoPE full attention, sliding-window attention, and gated linear attention hybrids on long-context tasks. Across models from 376M to 3B parameters, the study found trade-offs between length extrapolation and continued long-context training, and proposed sliding-window linear attention to improve extrapolation. The researchers also analyzed how position-biased and position-free attention modules cooperate.
机器之心 · 🔥 0
Experience and discussion from the community
Share my Sliding-Window Linear Attention experienceAsk about Sliding-Window Linear Attention
Nobody has shared their experience with Sliding-Window Linear Attention yet.