Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

iPrefBench tests how mobile agents infer preferences and ask users

AI summary

The iPrefBench paper introduces an executable iOS benchmark for evaluating whether personalized mobile agents can distinguish information available from user history from information that must be requested. It contains 130 underspecified tasks across 26 apps and 3,051 activity events for one persona, with every omitted requirement labeled history-inferable or ask-only. Evaluations of seven frontier agents found action completion as high as 91.6% but full task success as low as 47.7%, with agents especially weak at requesting missing information.

In-depth insight

The benchmark tests a distinct capability behind mobile automation: completing the right action for the right user while avoiding unsupported assumptions and unnecessary questions.

Computer Science > Computation and Language

arXiv:2610.34735 (cs)

[Submitted on 26 Oct 2026]

Title:iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents

Authors:Yuting Ning, Ruijia Cheng, Sushobhan Ghosh, Peitong Duan, HungShi Lin, Abhishek Sundararajan, Shreyash Pandey, Huan Sun, Jeffrey Nichols, Eldon Schoop

View a PDF of the paper titled iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents, by Yuting Ning and 9 other authors

#iPrefBench#Claude Opus 5#Claude Sonnet 5#Qwen3.8-27B#iOS

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.