A Practical Framework for Evaluating AI Agents: Three Layers and Eleven Dimensions
A Juejin article lays out an agent evaluation framework built on two questions: what to evaluate and how to evaluate. It divides evaluation targets into three layers (end-to-end, process, capability) and organizes metrics into an eleven-dimension taxonomy covering task success, tool use, planning, safety, efficiency, multi-agent collaboration and long-term memory consistency. It also cites ClawBench results in which the best of eight agents, Claude Sonnet 4.6, reached only a 33.3% task completion rate on everyday online tasks, while GPT-5.4 scored 6.5%, and it closes with a minimal Python script for computing task success rate and trajectory quality.
The article gives practitioners a concrete, runnable starting point for judging whether an agent actually completes tasks rather than merely producing plausible text, while the low completion rates it reports from ClawBench suggest vendor demos may overstate real-world agent performance.
核心结论:Agent 评测体系(Agent Evaluation)就是一套「系统地判断一个 Agent 好不好用」的方法,核心是回答两个问题:评什么(评测对象)和怎么评(评测维度与指标)。评什么分三层:端到端评测(把 Agent 当黑盒,只看任务有没有完成)、过程评测(看中间轨迹——规划、工具调用、反思是否合理)、能力评测(拆开测单项能力——工具使用、记忆、多 Agent 协作)。怎么评有一个行业共识的十一维框架:任务成功、输出质量、规划推理、不确定性决策、工具使用、自主性、学习适应、安全对齐、效率、多 Agent 协作、长期记忆一致性。现实很骨感:ClawBench 实测 8 款 Agent 处理日常在线任务,最好的 Claude 也才 33.3% 任务完成率——Agent 评测体系的存在,正是因为 Agent「看着能干活、实际经常干不完」。
#ClawBench#τ-Bench#AgentGym2#SlackBench#EnterpriseClawBench#Claude Sonnet 4.6#Qwen 3.5#GLM-5#GPT-5.4#NVIDIA