AWS tutorial connects decision models to AgentCore Evaluations
AWS Machine Learning Blog published a tutorial showing how to connect Jev and Strands Decider 2B to Amazon Bedrock AgentCore Evaluations through code-based evaluators. The sample converts OpenTelemetry trace and session data into model requests, then maps calibrated probabilities back to groundedness, helpfulness and goal-completion results. AWS says the decision models can run alongside 15 built-in evaluators, custom LLM-as-a-judge evaluators and deterministic checks.
The tutorial provides a practical pattern for adding narrow, thresholdable model checks to agent evaluation pipelines without replacing built-in evaluators.
Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a structured state and a fixed set of allowed answers and return a calibrated probability over them, rather than free-form text whose confidence drifts from run to run. That makes them a strong fit for the scoring step in an agent evaluation, where you want a label and a number you can threshold and trust.
Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, AgentCore Evaluations supports custom evaluators that you define as either an LLM-as-a-judge or a code-based evaluator. Custom LLM-as-a-judge evaluators apply your instructions and rating scale with a model in Amazon Bedrock. Code-based evaluators run your own logic in an AWS Lambda function, and that logic can call your choice of external system, model, or API, including a decision model.
#Amazon Bedrock AgentCore Evaluations#Jev#TypeSafe#Strands Decider 2B#AWS Lambda