Flo Health fine-tunes open models with Databricks to boost medical safety and usefulness
A Databricks blog post describes how Flo Health, a women's health app with 81 million monthly active users, built safer and more useful custom open models on Databricks AI Runtime. Flo first created an evaluation system of 26 AI judges (12 medical-safety, 6 micro-behavior, 8 usefulness) to turn medical standards into a training signal, generated more than 15,000 privacy-first synthetic questions, then ran supervised fine-tuning and DPO on 32 NVIDIA H100 GPUs using LLM Foundry and Compose-RL with MLflow tracking. Flo reports medical safety rising from 34% to about 61%, usefulness from 42% to 76%, and the share of responses meeting both from 16% to 47%, and estimates tailored models can cut inference costs by an order of magnitude versus frontier APIs; it plans to apply the pipeline to newer open-weight models such as Gemma, Qwen and Kimi.
It outlines a reusable post-training recipe that uses expert-calibrated AI judges and synthetic data to encode high-stakes safety standards directly into open-weight models.