SafeAgents locally screens coding-agent tool calls
SafeAgents is an Apache 2.0-licensed local proxy that checks coding-agent tool calls before the harness receives them. It combines two request-based checks with about 110 deterministic atif-scan detectors to identify undeclared tools, unexpected hosts, sandbox side channels, network bypasses, namespace escapes and reward-hacking attempts. An optional OpenAI-compatible judge can assess milder findings and may run locally. Flagged calls are replaced with a text notice, while --log-only mode records attempts without altering execution. The project supports OpenAI Chat Completions and Anthropic Messages, provides setup guides for 12 harnesses and reports end-to-end CI tests for six.
SafeAgents adds a local enforcement layer for tool calls, including calls made during reinforcement-learning rollouts and evaluations. The report says log-only mode can measure reward-hacking attempts, while blocking mode prevents flagged calls from running.
SafeAgents is a small local proxy that sits between a coding agent and its model server. It checks each tool call in the model's response before the harness receives it. A call it flags is replaced with a short text notice, so the harness gets a plain text reply and runs nothing.
You point the agent's base URL at the proxy instead of the model server. All checks run locally on your machine: the proxy only forwards traffic to the model server you already use, and doesn't send your prompts, code or tool calls anywhere else unless you point the optional judge at a remote server. Each tool call goes through up to three checks:
Catching reward hacking in RL and evals
The same detectors cover the tool calls a policy makes to game its reward instead of solving the task: writing its own reward file, editing the verifier's tests, or patching pytest so everything passes. Since safeagents is just a proxy in front of an OpenAI-compatible server, it can sit in front of the inference server used for rollouts or eval runs. In --log-only mode it records these attempts without changing the trajectories, which gives a measure of how often a model tries to cheat. In blocking mode, the call is replaced with a notice and never runs.
Some practical details:
Next steps are a small fine-tuned decision model for malicious tool calls and a public benchmark for it. Reports of interesting catches and of false positives are both useful.
Apache 2.0, runs locally: https://github.com/huggingface/safeagents submitted by /u/HauntingMoment
[link] [comments] |
#SafeAgents#atif-scan#Hugging Face#Claude Code#OpenCode#Qwen Code#OpenHands#DeepSeek Harness#pi#llama.cpp#OpenAI Chat Completions#Anthropic Messages