It was shocking for me to see the current state of radiology scribe software. We watched extremely intelligent and trained individuals being forced to use 30 year old SEGA controllers to make changes. It honestly reminded me of macaque motor control experiments with joysticks back in neuroscience wet lab at Columbia.
It was such a treat to work with real radiologists to build a realistic, diverse, and rigorous medical benchmark. Certainly an unfair advantage when radiology runs through
@nicochristie’s family tree. We had originally assumed that hillclimbing this would be trivial, but there is a lot of fun and nonstandard optimizations that one can make when speed AND accuracy are both prized. In fact, in the month that we had worked on this, we got bitter lessoned twice (generalist multimodal LLMs are better at technical transcriptions than specialized STT models, and then Opus 5.5 not giving a shit about how bad the transcription is provided it is somewhat phonetically intact)
Starting to learn about RIS and HL7 etc. Reach out to us if you’d like to chat, partner, or integrate this into your practices