Numan's Nu health coach puts evals first on LangGraph and LangSmith
Numan, a UK digital health company, describes how its AI health coach Nu runs as a LangGraph graph in which a supervisor model hands conversations to domain-specialist sub-agents, with evaluation handled in LangSmith. Numan says single-turn static fixtures and pre-scripted multi-turn conversations both failed to assess coaching dialogues, so it scores per category and has prototyped simulated patient personas, in which a cheaper model plays the patient and a judge scores the full transcript. The post also details safety probes that gate releases, in-graph guard nodes that review each outgoing reply against clinical safety rules, and daily human review of sampled conversations, all aimed at keeping Nu a wellness product under MHRA guidance rather than a medical device.
Numan says a regression in its safety probes automatically blocks a change from shipping, an example of eval criteria acting as a release gate near a regulated boundary.
Numan is a UK digital health company helping patients manage long-term conditions across weight, hormones, and sexual health. Many of our patients are on a weight-loss programme and use the app daily to track progress.
When building Nu, our AI health coach, one of our design challenges was delivering high-quality health coaching without crossing the line into medical diagnosis, prescriptions, or making clinical decisions—all strict boundaries for agents in our highly regulated medical space. While we initially focused on the agent and automation around it, what turned out to matter most was the quality of our evaluation criteria. This is the story of how we developed and improved our evaluation criteria to deliver high-quality user experiences while steering clear of areas subject to regulation.
Where standard eval approaches fell short for Nu
We started where most teams start, with a dataset of inputs and expected outputs, and an LLM judge scoring each reply. While that playbook works well for a Q&A agent, coaching is a conversation, not an answer. Every approach that fixed Nu's reply in advance failed in two ways: