AWS 开源通用智能体评估框架 UAEF
亚马逊云科技(AWS)开源了通用智能体评估框架 UAEF,该系统通过适配器把 LangGraph、LangChain、Strands Agents、Amazon Bedrock AgentCore 等不同框架的执行轨迹统一为一种格式,再以 36 项内置指标、7 个质量维度打分。评估结果存入 Amazon DynamoDB 与 Amazon S3,可生成统计报告,并可选用 Amazon Bedrock 上的模型生成分析洞察。据 The Next Web 报道,单项得分下降超过 5% 会被标记为回归,AWS 称已有电信、医疗和金融服务领域的 12 家机构用它在发布前比较智能体版本。
UAEF 为不同智能体框架提供统一的评估层,团队更换框架时不必重建测试,并且可保存运行结果做版本对比,在发布前发现退化问题。
Teams building AI agents on AWS use a range of frameworks, including LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore. Each framework records agent execution in its own way, so the structure of messages, tool calls, and handoffs between agents differs from one to the next. Evaluation tooling tends to follow the same boundaries: teams build metrics, scoring logic, and result storage against a single framework’s trace format, and they must rebuild them when they switch frameworks or run agents on more than one.
This coupling makes a basic development question harder to answer: did a change to the prompt, model, or tools improve the agent? Answering it requires consistent metrics across versions of an agent and a shared record of past results to compare against. Agent quality also spans several dimensions that a single accuracy score does not capture: tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination, and reasoning.
In this post, we introduce the Universal Agent Evaluation Framework (UAEF). Customers across 12 organizations in the telecom, healthcare, and financial services industries used UAEF to evaluate their AI agents with one consistent set of metrics and compare agent versions before release. UAEF is a framework-agnostic evaluation system: it uses adapters to convert traces from each framework into a common format, then scores those traces with 36 built-in metrics across 7 dimensions. It stores the results in Amazon DynamoDB and Amazon Simple Storage Service (Amazon S3) and generates reports and insights that inform the next iteration of the agent. This post describes how UAEF works, provides a playbook for choosing metrics and preparing datasets, and points to runnable notebooks for each supported framework.
#Amazon Web Services#Universal Agent Evaluation Framework (UAEF)#LangGraph#LangChain#Strands Agents#Amazon Bedrock AgentCore#Amazon Bedrock#Amazon DynamoDB#Amazon S3#Langfuse#The Next Web