FDE-Gym benchmark: agents score 0.56 on real-world delivery tasks
A new paper listed on Hugging Face's Papers feed introduces FDE-Gym, a benchmark of 95 end-to-end delivery tasks drawn from 20 customer scenarios across 18 sectors and adapted by industry experts from real delivery projects. In simulated customer environments, agents interact with scripted stakeholders and work on production-like systems, scored by how much they improve the customer's own success metric (0 for an unchanged system, 1 for a reference solution), with penalties for messages, trial runs and waiting time. The authors report that in the realistic setting, where unstated constraints must be discovered, the best configuration reaches a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks; even on the 23 tasks where it obtained every key fact from stakeholders, only 10 reached a normalized business uplift of 0.8 or above.
It shifts evaluation from well-specified tasks toward the ambiguity and hidden constraints of real delivery work, suggesting agents' bottleneck may lie as much in turning information into a working solution as in finding it.