DeepEval

Headlines · 3

1. 2h
   A Practical Guide to AI Agent Evaluation Toolchains

   A Juejin tutorial organizes AI agent and application evaluation into four approaches: manual review, custom scripts, evaluation frameworks, and managed platforms. It compares tools including Promptfoo, DeepEval, LangSmith, RAGAS, and Langfuse, while advising teams to define datasets, metrics, and scorers before selecting execution tools. The article also warns about licensing, judge-model costs, vendor lock-in, and the limits of automating poorly defined evaluations.

   掘金 · 人工智能 · 🔥 5
2. 4h
   AWS introduces Amazon Bedrock AgentCore Evaluations

   AWS Machine Learning Blog introduced Amazon Bedrock AgentCore Evaluations, combining AWS-built-in evaluators with DeepEval and AutoEval metrics for measuring AI-agent behavior. The service supports on-demand evaluation for direct session testing, online evaluation of sampled production traffic, and batch evaluation against fixed session datasets, while also offering custom LLM-as-a-judge evaluators. AWS states that its built-in evaluators are quality-tested and benchmarked for reliability, though the evidence comes from the vendor's own tutorial.

   AWS Machine Learning Blog · 🔥 9
3. Nov 1
   A Task-Level Framework for Evaluating AI Agents

   The ETF Agent evaluation framework argues that a green test only confirms that a check passed under defined conditions, not that the user’s task was completed. It recommends separating runtime behavior, evidence retrieval, content judgment and user outcomes, while preserving failed, inconclusive and corrected observations so model and system quality can be compared on a valid denominator. The article combines practical guidance on RAG, traces, model judges, sub-agent contributions and cost with historical examples from its own development process.

   掘金 · 人工智能 · 🔥 0

Experience and discussion from the community

Share my DeepEval experienceAsk about DeepEval