Speculative execution cuts on-device voice agent latency from 5.79s to 4.60s
An arXiv paper proposes speculative tool execution for on-device cascaded voice agents: a Predictor module anticipates tool calls from partial ASR hypotheses, runs them speculatively and caches the results, which are then injected into the LLM prompt for faster responses. A rule-based validation step filters cached results invalidated by user self-corrections, and the LLM can still issue tool calls directly, so worst-case latency stays bounded by the serial baseline. In live measurements on a fully implemented Android voice assistant, median time-to-first-audio fell from 5.79s to 4.60s and the standard deviation from 3.49s to 2.81s.
It shows a scheduling-and-caching route to lower voice-assistant latency without changing the underlying models.