Attention Residuals
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 3
- OASIS adds null routing to attention residuals
A team from Northwestern University, the Illinois Institute of Technology, Rutgers University, and other institutions proposed OASIS, whose token- and depth-level null options let Attention Residuals abstain from assigning full probability mass to real branches. According to Xinzhiyuan’s report on the paper, the method was accepted to NeurIPS 2026 and reduced average activation kurtosis by 95.90% versus Vanilla AttnResidual across LLaMA-3.2-1B, Qwen3-0.6B, and Phi-4. The reported experiments also found a 13.21-point Qwen3-0.6B W4A4 GSM8K Pass@1 advantage over Vanilla AttnResidual and recovered 77.3%–91.8% of selected RULER and LongBench performance losses at 12K-token contexts.
新智元 · 🔥 0 - 17-year-old ex-Kimi researcher Nathan Chen reportedly joins Discovery Loop
On October 20 the anonymous account Arfur Grok, which tracks Silicon Valley AI talent moves, reported that 17-year-old researcher Chen Guangyu (Nathan Chen) had joined Discovery Loop, the startup founded by former Google researchers including Jeff Dean. Chen had announced on October 15 that he was leaving Moonshot AI, where he worked on the Kimi models, and he publicly denied speculation that he was joining Anthropic. Neither Chen nor Discovery Loop has confirmed the move, and Discovery Loop said it does not comment on personnel matters. Founded in August 2026 by four former Google researchers, Discovery Loop has released no public product and is reportedly seeking funding at about a $50 billion valuation, with the goal of using AI to propose hypotheses, design and run experiments and iterate on results.
InfoQ 中文 AI&大模型 · 🔥 0 - 17-year-old Kimi architecture co-author Nathan Chen leaves Moonshot AI for Bay Area
On October 15, 17-year-old Nathan Chen announced on X that he had left Moonshot AI's Kimi team, calling the past year the best of his life so far. He joined in November 2025 as a 16-year-old intern and is a co-first author of the Attention Residuals paper, work that reports say is used in the Kimi K3 architecture. Anthropic researcher Sholto Douglas invited him to San Francisco in the comments, and Chen replied that he is heading to the Bay Area, but when asked whether he was joining Anthropic he answered "no"; his next employer, role and start date remain unknown. DoNews headlined the move as an Anthropic poaching, which conflicts with Chen's own denial.
36氪 人工智能 · 🔥 0
Experience and discussion from the community
Share my Attention Residuals experienceAsk about Attention Residuals
Nobody has shared their experience with Attention Residuals yet.