CritPt v2

Headlines · 4

1. 3d
   Artificial Analysis v5.0 evaluates real-world AI tasks

   A Juejin commentary describes Artificial Analysis Intelligence Index v5.0 as a framework for evaluating end-to-end workplace tasks rather than isolated question-answering. It says the index contains nine evaluations grouped into four equally weighted categories—knowledge work, coding, science and general capabilities—while the individual evaluations have different weights. The article highlights constrained workflows, deliverable quality, tool use, safety guardrails and first-attempt success, and notes limitations including an English focus, private test sets, high difficulty and limited multimodal coverage. These details reflect the commentary's interpretation and are not independently corroborated in the supplied material.

   掘金 · 人工智能 · 🔥 0
2. 7d
   Artificial Analysis Updates Six Capability Indexes to v2.0

   Artificial Analysis said it updated its six Capability Indexes to v2.0, aligning them with evaluations from the Artificial Analysis Intelligence Index v5.0 and placing greater emphasis on Non-Hallucination and Agentic Tool Use. The indexes map O*NET occupational tasks to representative benchmarks, weighting each benchmark by how often the capability appears in the occupation. They cover Finance & Accounting, Strategy & Ops, Legal, Healthcare & Medical, Engineering, and Economics. The reported changes include upgraded HLE and AA-Omniscience versions, removal of GDPval-AA from Agentic Knowledge Work, a new Science capability for Engineering using CritPt v2 and Terminal-Bench-Science 0.1, and FrontierCode 2.0 for Agentic Coding. The report said Claude Opus 5.5 (max) leads all six indexes. These details and rankings come from a single supplied report.

   ArtificialAnlys (X) · 🔥 0
3. 7d
   Artificial Analysis plans API release for improved CritPt v2

   Artificial Analysis said it worked with the CritPt authors to finalize the CritPt v2 grading server and planned to make it available through its API the following day. The authors improved the robustness of the grading-server logic, and Artificial Analysis incorporated those changes in version 5.0.1 of its Intelligence Index. It then regraded every model, finding small score increases for most models while frontier-model rankings remained unchanged, with scores moving by at most one point.

   ArtificialAnlys (X) · 🔥 0
4. Oct 26
   CritPt v2 benchmark released for research-grade physical reasoning

   DoNews reported the release of CritPt v2 as an evaluation dataset for research-grade physical reasoning in large language models. The supplied report provides no details about the dataset’s contents, evaluation methodology, participating models, or results.

   DoNews · 🔥 0

Experience and discussion from the community

Share my CritPt v2 experienceAsk about CritPt v2