AI Scientist

Headlines · 2

1. 7h
   Nature: AI agent scores just 2/6 and 1/6 on 'shadow evaluation' research test

   An arXiv preprint led by Princeton's Sayash Kapoor proposes "shadow evaluation": an AI system was pointed at the research questions of two papers submitted to NeurIPS, told to run its own project and write a paper, and then graded by the original authors. The agent harnessed Claude Opus 4.8 inside a modified OpenClaw scaffold with six days and $3,000 of compute credits per paper; it ran hundreds of experiments without stalling, produced plausible literature reviews and caught its own hallucinations without reward hacking, yet earned only 2/6 and 1/6 from the authors. The authors cite premature lock-in on a hypothesis, insufficient backtracking, overly lenient self-review and messy write-ups; Kapoor says full automation of open-ended research is far off, while AI Scientist co-creator Cong Lu called it an interesting data point but said the guardrails were too weak.

   虎嗅 AI · 🔥 9
2. 7d
   Sakana AI's MASS lets a model self-improve, but blind spots compound

   Researchers from Sakana AI and UC Berkeley introduced "Multi-Agent Self-Supervision" (MASS), in which a single language model acts as solver, evaluator and optimizer to improve itself recursively without a stronger teacher model. Using Qwen3.6-27B with Qwen Code, its win rate against the base model rose from 53.9 percent after one round to 69.9 percent after two, the self-evaluator's agreement with external judges climbed from 73 to 93 percent, and it reached 1.2 to 1.6 times the base model's score per output token on MLR-Bench, DSBench, ScienceAgentBench and AstaBench. The paper warns, however, that the internal evaluator sometimes accepted manipulated or invalid results as successes and baked them into the next model, a shared blind spot the authors call the potential "Achilles' heel" of recursive self-improvement and a reason to demand independent checks, limits on autonomy and clear conditions for intervention.

   The Decoder · 🔥 0

Experience and discussion from the community

Share my AI Scientist experienceAsk about AI Scientist