TL;DR: Clinical AI benchmarking has a core problem — real EHRs can't be shared and their labels aren't verifiable. This paper solves it with a fully synthetic hospital, and frontier models still fall short of top physicians. Title: Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark URL: Points 🏥 Built from medical education materials into 1,268 patients and 5,602 encounters, fully synthetic and PHI-free so it can be shared openly 🔗 Diagnoses, findings, and temporal relations are deterministically grounded in ICD-10-CM/SNOMED CT/LOINC, with labels derived mechanically from a knowledge graph 👨‍⚕️ Physicians distinguished synthetic from real charts at just 53% accuracy — essentially chance 📊 Across 10 models on 5 tasks, the best patient-diagnosis score was Kimi 2.5-thinking at 0.732 severity-weighted F1, matching average physician performance ⚠️ Still well below top physicians (0.89); every model missed roughly half the findings in summarization tasks 🔁 Swapping the generator model changed scores by ≤0.05, confirming it measures real clinical state, not generation artifacts Feels significant to finally have a benchmark that can measure clinical LLMs honestly, without a privacy tax. #MedicalAI# #LLMBenchmark#

Computer use is GA on the Claude API. Stellar production writeup by @davilocatelli13 at Asteroid: agents driving EHRs and payer portals, human steering down from 8.9% of runs to 0% since April.

NEW ON THE POD: I sat down with Yair Saperstein, MD MPH — co-founder and CEO of AvoMD — fresh off the company's Series A close in Q1. Links to listen (plus more details below): Listen on Apple: Listen on Spotify: Yair makes the case that the real prize in clinical AI isn't the AI scribe; it's the operating system living inside the EHR itself. Yair also shares why he thinks most of primary care ultimately gets replaced by AI voice chatbots, at-home labs, and wearables. I push Yair on the Open Evidence valuation question (politely sidestepped), how AvoMD sells differently to academic systems versus rural hospitals, and the "avoid the 800-pound gorilla" go-to-market strategy against Epic. Plus: what Dr. Oz told a ViVE crowd about agentic AI being in front of every patient by 2028 — and a closing piece of life advice that somehow involves Frogger. TIMESTAMPS [02:00] Why "efficiency" and "quality" in medicine became contradictions Yair couldn't ignore (and why that became Avo) [09:00] The three pitches Avo runs depending on who's in the room — academics get LOS and CDI, rural gets all-in-one, everyone gets Ask Avo [20:00] How to compete with the 800-pound gorilla without getting stomped (yes, Yair goes to UGM and sits in the front row) [31:00] Is Open Evidence overvalued? [36:30] The take that'll get him quoted out of context — most of primary care gets replaced by AI voice and at-home labs, and what that means for EHRs and payors [42:30] The Dr. Oz tell from ViVE: agentic AI in front of every patient by 2028, plus what FDA's January CDS rule shift actually means @YairSaperstein