OASIS adds null routing to attention residuals
A team from Northwestern University, the Illinois Institute of Technology, Rutgers University, and other institutions proposed OASIS, whose token- and depth-level null options let Attention Residuals abstain from assigning full probability mass to real branches. According to Xinzhiyuan’s report on the paper, the method was accepted to NeurIPS 2026 and reduced average activation kurtosis by 95.90% versus Vanilla AttnResidual across LLaMA-3.2-1B, Qwen3-0.6B, and Phi-4. The reported experiments also found a 13.21-point Qwen3-0.6B W4A4 GSM8K Pass@1 advantage over Vanilla AttnResidual and recovered 77.3%–91.8% of selected RULER and LongBench performance losses at 12K-token contexts.
OASIS offers a specific routing mechanism for reducing attention sinks and activation outliers while improving reported low-bit quantization and long-context performance.
新智元 2026-10-23 09:00 北京
新智元报道
增强模型的表达能力，可以让Transformer更灵活地读取和整合信息。Attention Residuals（AttnResidual）沿着这个方向，将注意力从token之间的信息交互扩展到网络深度，让每个子层学习选择哪些历史层输出。
但新增的选择能力也带来了新的问题。在研究团队评估的AttnResidual设置中，模型对初始token的注意力集中更加明显，激活异常值也显著增大。这些变化在低比特量化和长上下文评测中，进一步表现为性能退化。
来自西北大学、伊利诺伊理工学院、罗格斯大学等机构的研究团队提出OASIS，为这种双层信息路由提供显式的null选项，并用token层的null概率调整depth层的分支选择。该研究已被NeurIPS 2026接收。
#OASIS#Northwestern University#Illinois Institute of Technology#Rutgers University#Attention Residuals#LLaMA-3.2-1B#Qwen3-0.6B#Phi-4#RULER#LongBench