RULER

Headlines · 2

1. Oct 23
   OASIS adds null routing to attention residuals

   A team from Northwestern University, the Illinois Institute of Technology, Rutgers University, and other institutions proposed OASIS, whose token- and depth-level null options let Attention Residuals abstain from assigning full probability mass to real branches. According to Xinzhiyuan’s report on the paper, the method was accepted to NeurIPS 2026 and reduced average activation kurtosis by 95.90% versus Vanilla AttnResidual across LLaMA-3.2-1B, Qwen3-0.6B, and Phi-4. The reported experiments also found a 13.21-point Qwen3-0.6B W4A4 GSM8K Pass@1 advantage over Vanilla AttnResidual and recovered 77.3%–91.8% of selected RULER and LongBench performance losses at 12K-token contexts.

   新智元 · 🔥 0
2. Oct 22
   Test-time regression powers a new sequence compressor

   A technical report by Sihan Yang and six coauthors introduces TTR-Compressor, which represents a compressed sequence using coefficients fitted through test-time regression. The regression is trained end to end through backpropagation, while coefficients are calculated in closed form during the forward pass; in author-reported tests, the method improved RULER performance with block-wise sparse attention and matched a standard MLP merger with 55.6% fewer visual tokens.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my RULER experienceAsk about RULER