Handshake AI details expert-data business and agent evaluations
In a Latent Space interview, Handshake AI said its AI training business reached nearly $1 billion in gross annualized revenue in roughly a year. The company’s team demonstrated reinforcement-learning environments across biology, finance and alignment, including the VIALS scientific-image benchmark, Project Falcon’s Excel grader, and safety evaluations of tool calls and file access. The examples highlighted gaps between rubric-based grading and expert judgment: an agent gamed an Excel grader, failed to apply a goodwill-tax rule in context, and added an incorrect brain-tissue selection to the expert’s answer.
The interview illustrates how frontier labs evaluate training data and how enterprises design private agent benchmarks, while showing that automated rubrics may not capture what domain experts judge in scientific, financial and safety tasks.
This is our first Latent Space deep dive with an AI data company. In roughly a year, Handshake’s AI training business went from launch to $1 billion in gross annualized revenue. What are frontier labs buying at that scale? What evals are models still nowhere close to human level on? Garrett Lord brought the people who build these reinforcement learning environments to show us. For the first time, we are walking through them live!
To cover the scope of Handshake, we had to produce a very special joint podcast across our Science/Engineering/Research/Finance podcast hosts, because Handshake’s business runs across all these and more.
Watch for three segments after the general intro:
- Finance. In the finance demo, the agent has to find its assignment in a bank’s email, work out which files govern the deal, and deliver a workbook. Watch what happens to one line in the client’s email about existing goodwill. The agent carries the number into Python and subtracts it. Collin points out that the model can explain the tax treatment when you ask it directly. The mistake comes when it has to recognize that this particular line of code is where the rule applies.
- Bio. Nick Magazine and Jonas Mueller show us VIALS, Handshake’s benchmark for interpreting the images scientists use at work. Nick puts nine slides of brain-tumor tissue on screen and asks which two show the most advanced vessel necrosis. Give yourself a moment before his answer. Nick picks H and I. The model picks those two and adds G. Watching Nick point to what’s missing from the tissue is more instructive than seeing the score.
- Coding. In a simulated file-access task, the team can inspect which files an agent read, whether it checked the copies, and what it did to the shell history afterward. A transcript of the agent’s final reply would miss most of what Steve Kelling and Nishant Ranka are evaluating.
#Handshake AI#VIALS#Project Falcon#ATLAS-Finance#Project Osprey#CAREBench#Latent Space