MIT's HugAgent tests whether AI 'digital twins' can predict people in new scenarios
Researchers at MIT and other institutions propose HugAgent, a benchmark that uses interviews with 54 real participants to test whether large models can portray a specific person—not just restate current views but predict how that person would answer previously unseen hypothetical scenarios on universal healthcare, public surveillance and housing densification, across 1,742 evaluation items; the paper was accepted as an EMNLP 2026 Main Oral. Models did best at inferring a person's current beliefs (top accuracy 77.56%; GPT-4o 74.66%, against an 84.84% two-week self-consistency ceiling), while on the belief-dynamics update task the best result was 68.61% (Claude Sonnet 4.5), then LLaMA 3.3 70B at 67.57% and GPT-4o at 63.11%, against an 85.66% human ceiling. Personal interviews clearly helped prediction (GPT-4o rose from 39.83% to 63.11%), but adding more interview rounds did not consistently improve forecasts for new situations. Why it matters: It shows that piling up interview data about a person does not by itself make a model a reliable stand-in for them in unfamiliar situations—a key limitation for synthetic respondents and personalized agents. What do you think?
Open the headline: source, AI briefing and more →Comments
…