Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

OpenAI Proposes Perplexity as a Shared AI Progress Metric

AI summary

OpenAI Chief Scientist Jakub Pachocki published internal pretraining perplexity measurements for OpenAI models dating back to GPT-3.5. He argues that perplexity, or cross-entropy loss, could help compare model developers and inform shared safety standards. Pachocki said OpenAI accepted the risk of giving competitors a clearer picture of its progress because the urgency of establishing shared industry safety measures outweighed that concern. He also projected that further reductions in log-perplexity could be associated with new capabilities, while acknowledging that dataset selection affects comparability and algorithmic progress can blur the distinction between training phases.

November 22, 2026

ResearchSafety

Perplexity and Intelligence

By Jakub Pachocki, Chief Scientist at OpenAI

From afar, it might seem like modern AI is built similarly to how software has been built in previous decades: with a diverse set of techniques targeting specific capabilities. There are indeed a variety of methods, but rather than designing something from scratch, they are mostly harnessing the intelligence coming from a much simpler and less interpretable process: training a compressor (or, equivalently⁠(opens in a new window), predictor) on a very large corpus of data.1

I believe tracking the pace of progress on training better compressors is key to understanding the arc of advances in AI. In this post, I argue for establishing perplexity (or, equivalently, cross-entropy loss) as a metric to compare between model developers and inform shared safety standards. As a first step, I provide more transparency into pretraining progress at OpenAI. Publishing our internal perplexity measurements took some consideration; it gives competitors a clearer picture of where we stand, but that concern is outweighed by the urgency of establishing shared⁠(opens in a new window) safety measures across the industry. We at OpenAI hope for this to be a step towards building a shared scientific basis for future pacing efforts.

#OpenAI

Original · 7 min readRead the original
Group chat

No comments yet. Start the conversation.