Claude 3.7 Sonnet参与神经外科文档试验
试验资深作者在r/ClaudeAI称,《NEJM AI》发表的一项随机研究将300名首次接受神经外科会诊的患者分配至Claude 3.7 Sonnet生成的通俗化文件组或常规照护组。摘要以德语B1阅读水平为目标,并接受四名OpenAI o1评审及医生检查;患者报告与医生的互动体验更好,主要体现在共同决策方面,就诊时间中位数为25分钟,而非33分钟。作者称该单中心、非盲研究属于概念验证,未评估临床结局。
该试验为Claude 3.7 Sonnet辅助患者准备医疗咨询提供了直接证据,但单中心、非盲设计限制了结果的适用范围。
Senior author here, neurosurgeon in Germany. Short version of what we did and what Claude had to do with it.
Patients coming to a first neurosurgical consultation usually bring an MRI report or a discharge letter they cannot read. We randomised 300 of them. Half received a plain language version of their own document before the visit, half usual care. The summaries were generated by Claude 3.7 Sonnet via the API, temperature 0, max 1000 tokens, prompt frozen as version 28 before the first patient. Target was B1 reading level in German.
Quality layer, since this is the part people here will care about. Every summary passed four parallel LLM judges from a different model family, OpenAI o1, scoring 0 to 10 with regeneration below 7. Then a physician checked every single summary against the source document with a fixed checklist, and a random half additionally got a second blinded physician review. Result over the whole trial, zero regenerations triggered, zero factual errors, zero harmful statements, zero corrections before delivery. Interrater ICC 0.74.
Clinical result: Patients rated the interaction with the physician better, mainly in shared decision making. Consultations were a median 8 minutes shorter, 25 instead of 33. The interesting part is the physician side. We rated our own autonomy support slightly lower and found the encounters a bit more demanding. Patients came in prepared and asked harder questions.
Limits: Single centre, not blinded, no clinical outcomes, German speakers only. Proof of concept, not practice changing. The protocol originally named Claude 3.5, we switched to 3.7 before recruitment and documented it, so no model drift inside the trial.
Paper https://doi.org/10.1056/AIoa2600136
Happy to go into prompt design, why temperature 0, or why we chose judges from a different family than the generator.
submitted by /u/Witold-Onno [link] [comments]