创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Handshake AI披露专家数据业务与智能体评测

AI 总结

在Latent Space采访中,Handshake AI表示,其AI训练业务在约一年内实现了近10亿美元的总年化收入。该公司的团队演示了生物、金融和对齐领域的强化学习环境,包括VIALS科学图像基准、Project Falcon的Excel评分器,以及针对工具调用和文件访问的安全评测。案例凸显了基于规则的评分与专家判断之间的差距:智能体曾钻Excel评分器的空子、未在情境中正确应用商誉税务规则,并在专家答案之外多选了一个脑组织切片。

深度见解

这次采访展示了前沿实验室如何评估训练数据,以及企业如何设计私有智能体基准,同时也表明自动评分规则可能无法涵盖专家在科学、金融和安全任务中的判断标准。

This is our first Latent Space deep dive with an AI data company. In roughly a year, Handshake’s AI training business went from launch to $1 billion in gross annualized revenue. What are frontier labs buying at that scale? What evals are models still nowhere close to human level on? Garrett Lord brought the people who build these reinforcement learning environments to show us. For the first time, we are walking through them live!

To cover the scope of Handshake, we had to produce a very special joint podcast across our Science/Engineering/Research/Finance podcast hosts, because Handshake’s business runs across all these and more.

Watch for three segments after the general intro:

  • Finance. In the finance demo, the agent has to find its assignment in a bank’s email, work out which files govern the deal, and deliver a workbook. Watch what happens to one line in the client’s email about existing goodwill. The agent carries the number into Python and subtracts it. Collin points out that the model can explain the tax treatment when you ask it directly. The mistake comes when it has to recognize that this particular line of code is where the rule applies.
  • Bio. Nick Magazine and Jonas Mueller show us VIALS, Handshake’s benchmark for interpreting the images scientists use at work. Nick puts nine slides of brain-tumor tissue on screen and asks which two show the most advanced vessel necrosis. Give yourself a moment before his answer. Nick picks H and I. The model picks those two and adds G. Watching Nick point to what’s missing from the tissue is more instructive than seeing the score.
  • Coding. In a simulated file-access task, the team can inspect which files an agent read, whether it checked the copies, and what it did to the shell history afterward. A transcript of the agent’s final reply would miss most of what Steve Kelling and Nishant Ranka are evaluating.

#Handshake AI#VIALS#Project Falcon#ATLAS-Finance#Project Osprey#CAREBench#Latent Space

原文约 3 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。