A Task-Level Framework for Evaluating AI Agents
The ETF Agent evaluation framework argues that a green test only confirms that a check passed under defined conditions, not that the user’s task was completed. It recommends separating runtime behavior, evidence retrieval, content judgment and user outcomes, while preserving failed, inconclusive and corrected observations so model and system quality can be compared on a valid denominator. The article combines practical guidance on RAG, traces, model judges, sub-agent contributions and cost with historical examples from its own development process.
The framework provides AI-agent builders with a source-backed way to avoid turning local test success or an aggregate score into an unsupported claim about real task value.
绿灯只说明某项检查在给定条件下通过,不会自动证明用户的任务已经完成。 Agent 评测最重要的工作,是先确认检查了哪条边界、评的是哪份内容、分母包含哪些任务,再比较质量和成本。
这不是要求每次修改都重跑整套研究,而是让证据与声明相匹配:局部逻辑用局部检查,跨进程效果看真实交接,模型质量看实际产物。否则,一个准确的测试数字也可能支撑一项错误的产品结论。
我们早期遇到过这样的情况:服务端有候选答案,自动裁判给出了结果,运行状态甚至已经是 completed;继续沿前端链路检查,却发现候选存在、服务端发送、浏览器收到和展示,并不是同一个事实。
#ETF Agent#Research Runtime#OpenTelemetry#Langfuse#Ragas#DeepEval#PostgreSQL#Playwright#Vitest#AI SDK#MCP