AWS details a nine-stage AI agent evaluation framework on Bedrock AgentCore
An AWS Machine Learning Blog post describes how AWS's Sales, Marketing, and Global Services (SMGS) Operations team built a continuous quality-evaluation framework for a production multi-agent system on Amazon Bedrock AgentCore Evaluations. The framework spans nine lifecycle stages—defining quality, experimentation, CI merge checks, CD production gates, canary scoring, adversarial testing, live drift detection, a real-time pulse, and an automated feedback loop—and runs at three levels at once: tools, individual sub-agents, and multi-agent orchestration, each with its own golden set and promotion criteria. AWS reports that its internal sales agents produced fluent but wrong answers such as labeling $410.08 of three-month revenue as "$410.08 Million" and presenting estimates as sourced facts while conventional monitoring stayed green; in one pre-production test, mismatched data, tool, and prompt changes dropped the pass rate from 78 percent to 38 percent.
It turns the common worry that agents fail silently—invisible to latency and error-rate monitoring—into a reusable, continuously applied evaluation practice.
The North Star
Every organization racing to deploy AI agents faces the same reckoning: an agent that demos beautifully can quietly degrade in production, and by the time a human notices, the trust is already broken. Where traditional software fails loudly, agents fail softly. They return an answer that looks confident, reads fluently, and is wrong. When that answer drives a real business decision, the cost is not a broken feature. It’s a broken relationship with the user.
That’s the quality problem at its core. The challenge is therefore not building agents that work once, but proving they keep working, at scale, across every sub-agent, every workflow, and every release. The teams that win the agent era will be the ones who treat quality not as a final QA step but as the operating system of the agent itself: continuously measured, automatically defended, and improved with every interaction. In this post, we show how we built a continuous quality loop for production-ready agentic solutions at scale using Amazon Bedrock AgentCore.
Business problem
The solution uses Amazon Bedrock AgentCore and other AWS services for internal use, so sellers and sales leaders can ask business questions in plain English from Amazon Quick. Behind that single experience, several specialized sub-agents work together: the Performance Insights Agent for territory-scoped revenue and performance, the Customer Insights Agent for customer intelligence, the Data Insights Agent for text-to-SQL over foundational datasets, and direct dashboard access. Answers are scoped to each user’s territory and role-level security. Field teams use these answers to assist in planning account strategy, prepare for customer meetings, so the numbers must be right. The most damaging failure is not a crash. It’s a fluent, confident answer that is quietly wrong. Before launch, when we evaluated the Customer Insights Agent, we found exactly that:
#Amazon Bedrock AgentCore#Amazon Bedrock#Amazon Bedrock Guardrails#AWS Lambda#Amazon CloudWatch#Amazon Quick#Amazon S3#Amazon Redshift