Realtime-Bench tests AI systems under real-time interaction
Researchers introduced Realtime-Bench, a 50-task benchmark for evaluating general-purpose AI systems whose decisions and actions must respond to changing conditions in real time. It includes 30 web tasks and 20 game tasks, measuring temporal state awareness, adaptive deliberation, and time-aware action selection under wall-clock, paused-clock, and token-clock conditions. Import AI reported that GPT-6-Astra achieved the highest wall-clock pass rate at 36%, while high-reasoning models approached 70% to 80% with the clock paused.
Realtime-Bench offers a targeted way to assess whether AI agents can handle time-sensitive interactive tasks, where delayed decisions can directly cause task failure.
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.
LLMs are very bad at low-latency tasks: …Realtime-Bench says it’ll be a while till AI systems can play games or do online auctions for you… Researchers have built Realtime-Bench, “a suite of carefully curated dynamic tasks where delayed actions directly affect success”. The test helps evaluate how well modern, general-purpose AI systems can perform tasks where decisions need to be made in real-time. The benchmark consists of 50 interactive game and web tasks, split across 30 web tasks in four categories (travel, e-commerce, service, and security), and 20 game tasks (arcade, puzzle, strategy, and simulation). Who did the research: The benchmark was built by people at the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Steadyworks. What it measures: The test is designed to probe for three qualities: temporal state awareness (”RT-Perceive”), where models can use observations to track changes in the world; adaptive deliberation (”RT-Deliberate”), where models alter their reasoning length and depth according to available time; and time-aware action selection (”RT-Act”) where models need to appropriately time their actions for the situation. “For example, an agent purchasing a ticket must track the sale status and assess whether observed availability is still current (RT-Perceive), limit further deliberation when the purchase window is closing (RT-Deliberate), and choose between waiting for sales to open and purchasing while tickets remain available (RT-Act).”
#Realtime-Bench#NVIDIA#University of Texas at Austin#Ohio State University#University of Illinois at Urbana-Champaign#Northwestern University#Steadyworks#OpenAI#GPT-6-Astra#Anthropic#Google#DeepSeek