Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

UniPat AI leads launch of SPU wet-lab prediction benchmark

AI summary

According to a QbitAI report republished by 36Kr, UniPat AI led the launch of the Science Prediction Universe (SPU), a benchmark for evaluating AI systems that conduct autonomous research, use scientific tools and predict wet-lab outcomes. It contains 116 tasks and 4,207 experiments across chemistry, materials science, biology and medicine: 100 public tasks drawn from 85 recent papers and 16 private tasks based on unpublished collaborative experiments. The dataset and code have been open-sourced, while the public tasks are planned for release as complete Harbor task packages. Models receive eight hours per run, including an H20 GPU, and are scored by comparing predicted rankings with measured experimental rankings using Spearman correlation. The first evaluation covered 12 models in 13 configurations. Claude Opus 5.5 had the highest average score at 0.486 and exceeded the random threshold on about half of the tasks; the report said nearly all configurations exceeded that threshold on 34%–52% of tasks.

In-depth insight

SPU provides a unified way to measure AI systems on autonomous literature research, scientific-tool use, long-horizon execution and wet-lab ranking predictions. Its reported results also expose a substantial gap between current model performance and reliable experimental prediction, while providing researchers and model teams with a common metric for comparing and studying that gap.

不ber,现在的基准测试都进化到酱婶了???

这回不刷题了,就直接给AI一组实验条件,让它自己研究、提前猜真实湿实验的结果……

比如,同一个反应换几种条件,哪种能得到更多目标产物?一批抗体变体,哪个表达量更高?

答案先藏好,等AI交卷,再拿实测结果对一对。

你还别说,这事儿现在真值得一测。

这两年,AI for Science的方式越来越深入,AI已经能自主查文献、用软件,为完成一项分析任务连续做上几小时也不在话下。

#UniPat AI#Science Prediction Universe (SPU)#Claude Opus 5.5#GPT-6 Astra#Qwen3.8#Fable 5.1#Claude Opus 5#NVIDIA H20

Original · 13 min readRead the original
Group chat

No comments yet. Start the conversation.