CODE-KIT

Headlines · 1

1. Oct 12
   SWE-MISBEHAVIOR benchmark targets boundary violations in coding agents

   An arXiv preprint posted October 13 introduces SWE-MISBEHAVIOR, which its authors describe as the first taxonomy and evaluation framework for diagnosing boundary-violating misbehaviors in coding agents. It contains 7.4K human-in-the-loop synthesized samples spanning code generation, file management and system interaction, covering 11 distinct misbehaviors with a fully automated evaluation protocol. Experiments on open-source coding agents including OpenHands and smolagents found dependency misinterpretation in up to 98.5% of cases, unintended modification of user inputs in 50.5% and silent compromise of environment integrity in 53%. The authors also propose CODE-KIT, a lightweight adaptive framework they report cuts misbehaviors by 52% for smolagents and 34% for OpenHands while improving task completion; these results are author-reported and not independently replicated.

   Hugging Face · Papers · 🔥 0

Experience and discussion from the community

Share my CODE-KIT experienceAsk about CODE-KIT