τ-bench
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 3
- Yao Shunyu details foundation-model engineering priorities at Tsinghua
According to 36Kr AI, Tencent chief AI scientist Yao Shunyu held a closed-door exchange at Tsinghua University’s Yao Class on November 18, covering foundation-model infrastructure, training algorithms, data, safety, organization and commercialization. He questioned metric-driven evaluation and “horse-race” resource allocation, arguing that independent evaluations, data-quality judgment, engineering rigor and organizational trust remain central to model development. He also said concepts such as RSI should return to concrete engineering problems, while startups should reassess how their businesses relate to advances in foundation models.
36氪 人工智能 · 🔥 6 - A Practical Framework for Evaluating AI Agents: Three Layers and Eleven Dimensions
A Juejin article lays out an agent evaluation framework built on two questions: what to evaluate and how to evaluate. It divides evaluation targets into three layers (end-to-end, process, capability) and organizes metrics into an eleven-dimension taxonomy covering task success, tool use, planning, safety, efficiency, multi-agent collaboration and long-term memory consistency. It also cites ClawBench results in which the best of eight agents, Claude Sonnet 4.6, reached only a 33.3% task completion rate on everyday online tasks, while GPT-5.4 scored 6.5%, and it closes with a minimal Python script for computing task success rate and trajectory quality.
掘金 · 人工智能 · 🔥 0 - τ-Emotion-Bench tests tool-calling agents against emotionally varied users
An arXiv paper submitted on 14 October, “τ-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions,” introduces a ten-domain benchmark that evaluates tool-calling agents under controlled emotion-conditioned user interactions while keeping the underlying task objective unchanged. The authors argue that existing benchmarks largely assume cooperative, behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. A companion pipeline, Tracer, builds grounded, policy-valid tasks and collects trajectories through an agent-user-environment harness that lets dissatisfied users terminate inefficient interactions, filtering trajectories with excessive user dependence. The authors report that frontier agents remain unreliable under two user simulators, with substantial gaps between average task success and consistent task completion, and that Tracer-generated supervision improves Qwen3 models across scales and transfers to τ-Bench and τ-Trait. The paper was accepted to the UserSim Workshop at NeurIPS 2026.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my τ-bench experienceAsk about τ-bench
Nobody has shared their experience with τ-bench yet.