Claude 3.7 Sonnet Tested in Neurosurgical Patient Document Trial
A senior trial author reported in r/ClaudeAI that a randomized study published in NEJM AI assigned 300 first-time neurosurgical consultation patients to either Claude 3.7 Sonnet-generated plain-language documents or usual care. The summaries targeted a B1 German reading level, passed checks by four OpenAI o1 judges and physicians, and patients reported better physician interactions, mainly in shared decision-making; consultations had a median duration of 25 minutes instead of 33. The author described the single-center, unblinded study as proof of concept because it did not assess clinical outcomes.
The reported trial offers direct evidence on Claude 3.7 Sonnet’s use in preparing patients for medical consultations, while its single-center, unblinded design limits how broadly the findings can be applied.
Senior author here, neurosurgeon in Germany. Short version of what we did and what Claude had to do with it.
Patients coming to a first neurosurgical consultation usually bring an MRI report or a discharge letter they cannot read. We randomised 300 of them. Half received a plain language version of their own document before the visit, half usual care. The summaries were generated by Claude 3.7 Sonnet via the API, temperature 0, max 1000 tokens, prompt frozen as version 28 before the first patient. Target was B1 reading level in German.
Quality layer, since this is the part people here will care about. Every summary passed four parallel LLM judges from a different model family, OpenAI o1, scoring 0 to 10 with regeneration below 7. Then a physician checked every single summary against the source document with a fixed checklist, and a random half additionally got a second blinded physician review. Result over the whole trial, zero regenerations triggered, zero factual errors, zero harmful statements, zero corrections before delivery. Interrater ICC 0.74.
Clinical result: Patients rated the interaction with the physician better, mainly in shared decision making. Consultations were a median 8 minutes shorter, 25 instead of 33. The interesting part is the physician side. We rated our own autonomy support slightly lower and found the encounters a bit more demanding. Patients came in prepared and asked harder questions.
Limits: Single centre, not blinded, no clinical outcomes, German speakers only. Proof of concept, not practice changing. The protocol originally named Claude 3.5, we switched to 3.7 before recruitment and documented it, so no model drift inside the trial.
Paper https://doi.org/10.1056/AIoa2600136
Happy to go into prompt design, why temperature 0, or why we chose judges from a different family than the generator.
submitted by /u/Witold-Onno [link] [comments]