Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Sim-to-Real Gaps May Help Explain Uneven AI Progress

AI summary

An analysis argues that the sim-to-real gap may help explain why AI has made rapid progress in programming and mathematics while its real-world impact in many professions remains modest. Programming and mathematical reasoning can be replicated and verified in digital environments, whereas many white-collar professions involve people and physical environments that are difficult to simulate. The analysis uses robotics examples to illustrate the challenge. It also cites GDPval, where the best score reportedly rose from 47.6% for Claude Opus 4.1 when the benchmark was released to more than 85% for current leading models.

In-depth insight

The analysis offers a framework for distinguishing strong benchmark performance from broad professional transformation. It attributes the difference to simulation and verification being easier in some domains than in work involving people and physical environments; this remains the author’s interpretation rather than a demonstrated causal finding.

The last year has seen incredible progress in model capabilities in certain domains — most notably programming, mathematics, and cybersecurity. Yet the real-world impact of these breakthroughs has been surprisingly modest — especially outside of those professions.

Lawyers, architects, salespeople, college professors, and others are certainly using AI models in their work, but AI models don’t seem close to transforming those professions the way they seem to be transforming mathematics and computer programming. And even in the software industry, we’re not seeing the kind of mass unemployment many people expected we would see once AI models reached human capabilities.

A big question is why — and whether the impact might grow in the next few months or years. Many people expect massive AI-driven disruptions to begin hitting other professions in 2028 or even 2027.

I don’t think that’s going to happen, and a concept from robotics helps to explain why: the sim-to-real gap. I encountered this concept when I was editing my colleague Kai Williams’s excellent September article about the scramble to collect robotics training data.

Suppose you want to train a robot to walk. One option is to create a virtual environment with a simulated robot. Then use reinforcement learning to reward behaviors that help the robot move around. The robotics startup Figure used this strategy last year and it worked! Figure said that it collected “years worth of simulated demonstrations in a few hours.” When they transferred the trained model to a real robot, the robot was able to walk “with no tweaks or changes.”

#OpenAI#DeepMind#Figure#GDPval#Claude Opus 4.1#AlphaGo#AlphaZero#Lean#Bracket Bot

Original · 6 min readRead the original
Group chat

No comments yet. Start the conversation.