SafeAgents 在本地检查编程智能体工具调用
AI 总结
SafeAgents 是一个采用 Apache 2.0 许可的本地代理，会在编程智能体的工具调用到达执行框架前进行检查。它结合两项基于请求的检查和约 110 个确定性 atif-scan 检测器，识别未声明工具、异常主机、沙箱侧信道、网络绕过、命名空间逃逸及奖励作弊尝试。可选的 OpenAI 兼容模型可评估较轻的发现，并可在本地运行。被标记的调用会替换为文本通知，--log-only 模式则只记录尝试而不改变执行。项目支持 OpenAI Chat Completions 和 Anthropic Messages，为 12 个智能体框架提供设置指南，并称其中 6 个通过端到端 CI 测试。
深度见解
SafeAgents 为工具调用增加了本地执行层，包括强化学习 rollout 和评测期间的调用。原报告称，日志模式可用于衡量奖励作弊尝试，而阻止模式可防止被标记的调用运行。
SafeAgents is a small local proxy that sits between a coding agent and its model server. It checks each tool call in the model's response before the harness receives it. A call it flags is replaced with a short text notice, so the harness gets a plain text reply and runs nothing.
You point the agent's base URL at the proxy instead of the model server. All checks run locally on your machine: the proxy only forwards traffic to the model server you already use, and doesn't send your prompts, code or tool calls anywhere else unless you point the optional judge at a remote server. Each tool call goes through up to three checks:
Catching reward hacking in RL and evals
The same detectors cover the tool calls a policy makes to game its reward instead of solving the task: writing its own reward file, editing the verifier's tests, or patching pytest so everything passes. Since safeagents is just a proxy in front of an OpenAI-compatible server, it can sit in front of the inference server used for rollouts or eval runs. In --log-only mode it records these attempts without changing the trajectories, which gives a measure of how often a model tries to cheat. In blocking mode, the call is replaced with a notice and never runs.
Some practical details:
Next steps are a small fine-tuned decision model for malicious tool calls and a public benchmark for it. Reports of interesting catches and of false positives are both useful.
Apache 2.0, runs locally: https://github.com/huggingface/safeagents submitted by /u/HauntingMoment
[link] [comments] |
#SafeAgents#atif-scan#Hugging Face#Claude Code#OpenCode#Qwen Code#OpenHands#DeepSeek Harness#pi#llama.cpp#OpenAI Chat Completions#Anthropic Messages