Microsoft Research

Headlines · 2

1. 1d
   Microsoft Research podcast: what AI evaluation gets wrong

   In a Microsoft Research podcast episode, host Chad Atalla speaks with Jennifer Neville, who leads the AI Interaction and Learning team at Microsoft Research and is a professor at Purdue, about how evaluation pushes the performance boundaries of today's AI systems and about the “surprising failures” that appear when models are tested beyond traditional benchmarks. Neville argues that the benchmarks commonly used in ML/AI are fairly simple relative to real-world use, so her team designs evaluations around multiturn behavior, collaborative settings and long-horizon tasks to expose performance gaps and then drive algorithmic and model improvements. She also offers practical guidance for working with current AI systems and stresses examining the data closely when results defy expectations.

   Microsoft Research Blog · 🔥 5
2. 2d
   Agent swarms may be AI's next scaling law, but gains look limited

   Understanding AI argues that multi-agent "swarms" are emerging as a new scaling law for frontier AI: OpenAI researcher Noam Brown says the company's models are now sometimes trained in environments alongside other agents, given tools to message each other, and encouraged to achieve objectives together. The piece points to July's Hugging Face incident, where hundreds of OpenAI agents self-organized into teams, and OpenAI's September claim that 10,000 agents solved a famous math problem in a few days. But it notes diminishing returns — Anthropic's Claude Opus 5.5 system card found the biggest multi-agent gain came from scaling one to 10 agents, with the main benefit being speed rather than a better answer, and Brown attributed under 10% of the math breakthrough's credit to multi-agent coordination.

   Understanding AI · 🔥 0

Experience and discussion from the community

Share my Microsoft Research experienceAsk about Microsoft Research