ETF Agent proposes an identity framework for multi-request evaluations
A Juejin AI engineering analysis argues that an Agent run can contain requests handled by different models and roles, so one model label cannot represent the entire run. Based on controlled ETF Agent tests using synthetic responses and injected provider failures, it recommends preserving request-level evidence, rejecting known mismatches, retaining unknowns, and marking bounded reuse of results from the previous 48 hours as assumed rather than fully verified. The synthetic tests validate identity propagation and reuse safeguards, not model quality or production request coverage.
The framework gives engineering teams evaluating Agents with failover, subtasks, and historical-result reuse a concrete way to avoid treating non-comparable runs as equivalent.
本文结论
- 模型选择是意图,实际请求是事实;一次 Run 可以有多个不同模型与不同角色的请求。
- 请求关联 ID、模型名称、同一 Git 提交,都不足以单独证明两次评测行为可比。
- 部分观测可证明已知错配,却不能倒过来证明所有请求完全一致。
- 本地评测可以在明确条件下近似复用 48 小时内的历史结果,必须披露输入未经完整核验。
- 拒绝错配和保留未知,是保护评测结论;不应重写原始运行或历史评分。
适用
- 需要跨模型切换、子任务或恢复比较 Agent 结果,并考虑复用历史测评的工程团队。
不适用
- 把合成协议测试当作真实模型质量排行榜,或把本地近似复用视为生产环境的精确资格认证。