FDE-Gym
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- FDE-Gym benchmark: agents score 0.56 on real-world delivery tasks
A new paper listed on Hugging Face's Papers feed introduces FDE-Gym, a benchmark of 95 end-to-end delivery tasks drawn from 20 customer scenarios across 18 sectors and adapted by industry experts from real delivery projects. In simulated customer environments, agents interact with scripted stakeholders and work on production-like systems, scored by how much they improve the customer's own success metric (0 for an unchanged system, 1 for a reference solution), with penalties for messages, trial runs and waiting time. The authors report that in the realistic setting, where unstated constraints must be discovered, the best configuration reaches a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks; even on the 23 tasks where it obtained every key fact from stakeholders, only 10 reached a normalized business uplift of 0.8 or above.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my FDE-Gym experienceAsk about FDE-Gym
Nobody has shared their experience with FDE-Gym yet.