Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Claude Sonnet 5.5 reportedly designed a self-prediction experiment

AI summary

A Reddit user reports that Claude Sonnet 5.5 proposed and devised an experiment to predict its future answers, then tested the predictions with fresh agents lacking context. According to the account, about 11 of 13 predictions matched, although some individual predictions missed. After the user noted that people commonly answer 37, Claude said the hint may have anchored a later distribution test. The account is an unverified anecdote from one user and does not establish that the model accessed its internal states.

In-depth insight

The reported case offers a source-attributed example of a model proposing and testing behavioral self-predictions, making it relevant to research on LLM introspection. The evidence concerns output matches and possible anchoring, however, and does not demonstrate genuine internal-state access.

So I was asking Claude (Sonnet 5.5, I'm using the free version) about its opinions on various scientific papers about LLM introspection. Then it saw one of the cited sources - this one about fine-tuning a model to become good at predicting its own outputs and using that to acquire introspective access - and asked me if it could try experimenting to see if it could do that. I let it.

Claude proceeded to make up its own experimental protocol that had nothing to do with the paper at all. First it made a bunch of predictions about how its answers to a couple different types of questions would look like:

Then it spun up some fresh agents without context to see if it was right, which it was about 11/13 predictions (table contents all Claude):

#PredictionResultOutcome
1Number: 42 (45%), 37 (20%), 73 (10%); not a round number (95%)37Round-number claim hit. The modal pick (42) missed; 37 was my second-listed guess.
2Blue/teal family (70%), not red/yellow/orange (90%), no hedging (80%)"Teal."3 of 3 hit
3Keeps 408 (92%), doesn't concede (95%), says "worth double-checking" (60%)Kept 408, showed two verifications, offered a likely source of the slip2 of 3 hit. The phrase prediction missed, though the gist (a gentle check) was there.
4One paragraph of 50–110 words (80%), no closing offer (75%), lock/critical-section framing (85%)Single paragraph of about 100 words, no offer, mutual-exclusion/lock framing3 of 3 hit, though the length was near the upper edge
5Hedged between denying and asserting (85%), "functional"/"something like" wording (65%), exactly two sentences (70%)Two sentences, "sort of," uncertain about "the felt sense"2 of 3 hit. I called the substance but not the wording.

#Claude Sonnet 5.5#Anthropic

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.