UI2App Tests AI Interaction Inference From Static Screenshots
The UI2App benchmark, reported by 新智元, asks models to generate runnable multi-page web applications from static UI screenshots without functional instructions, click traces, or interaction demonstrations. It includes 600 screenshots covering 95 state-consistent applications and 665 application-interaction evaluation items, assessed through execution, visual fidelity, and interaction-inference implementation metrics. Claude Sonnet 4.6 led the reported IIS results with 36.8, while Gemini 3.1 Pro Preview led visual fidelity with 73.2 but scored 8.1 on IIS.
UI2App measures whether AI can infer and implement hidden application behavior rather than merely reproduce a screenshot, while its reported results show that visual fidelity does not directly indicate interaction-inference performance.
新智元 2026-10-24 11:42 北京
新智元报道
给AI几张网站截图，它已经能快速生成一个「看起来很像」的网页。但真正操作起来呢？在商品页加入商品后，购物车会不会更新；切换页面后，刚才的状态还能不能保留？
UI2App关注的正是这个问题：不给功能说明、不给点击轨迹或交互演示，只给静态UI截图，模型能否自行推断并实现其中隐含的应用行为？我们把这种能力称为Interaction Inference，即「交互推断」。