@UCSF surgeon-scientist. Clinical AI, computational genomics, health policy. Cofounder @memorahealth (acq). Alum @ycombinator w18, @broadinstitute, @harvardmed.
San Francisco, CA
Joined December 2012
- Tweets5.2K
- Following972
- Followers2.8K
- Likes17.4K
Pinned Tweet
1/ When expert clinicians see similar cases, they often reuse stable mental templates, known as diagnostic schemas.
I built a way to test whether LLMs do the same.
(Below: my study group thinking through a cardiogenic shock case during med school)
In Madrid for #ESMO26!
On Monday morning I'm giving a talk on clinical deployment of agents and evals in the Al-driven Remote Patient Monitoring session (10:15am, Hall 9). Come say hello!
1
2
417
Excited to try this (especially since OE has full text data)!
For the last couple months I’d been routing at least half of my clinical questions into the CLI with my evidence-grader skill for the exact same reason; it’s hard to gauge the quality of evidence in these answers short of reading the papers myself or uploading their full text into a model and asking for the appropriate study appraisal tool.
Some clinical questions have thirty years of randomized trials behind them. Some have four case reports.
Answers on OpenEvidence now carry a grade for how much certainty the underlying evidence can support, using the same GRADE criteria Cochrane and the WHO apply to guidelines. Study design, agreement across trials, precision, and whether the studied population resembles the patient in front of you.
@FierceHealth got the first look at EvidenceGrade. Link in the reply below.
3
786
Heading to Seoul for ICML!
Presenting a Spotlight paper on Saturday at the Structured Data for Health workshop.
If you work on AI for health/bio, AI safety, uncertainty calibration, or CoT faithfulness come say hello or DM me.
📍July 11, COEX Hall D2, 9:50-10:50AM
1/ When expert clinicians see similar cases, they often reuse stable mental templates, known as diagnostic schemas.
I built a way to test whether LLMs do the same.
(Below: my study group thinking through a cardiogenic shock case during med school)
11
1,193
In medicine, we’re already seeing startups expand into clinical decision support tools with read-access to the entire health record, which are often *decades* long per patient. Compressing not only text, but imaging, labs/trajectory, and reasoning in the least lossy way possible is critical, and representation-level tools may be the way to do it! Check out Vishnu’s work on the subject.
I've been researching how to compress coding agent context at the representation level, working directly with models' internal vectors instead of summarizing text. So far, I've hit 2x+ compression while preserving most factual recall.
2
34
6,868
Seeing a lot of bitter lesson/scale > specialized model tweets about this study, but short of OE/UTD intentionally buying weaker models, this result is more of a *harness* issue than a model one.
MedQA score ranges across models were within 9 points (so less likely a base knowledge gap); the gap between gen vs clinical models widened on the real world clinical question answers, where the problems were mostly in omission and disorganized outputs rather than wrong facts.
For medical information, general AI frontier models (Google, OpenAI, Anthropic) outperformed specialized @openevidence and @UpToDate as assessed by 12 US clinicians, randomized and blinded to which model and extensive testing/benchmarks. This was not anticipated. @NatureMedicine
nature.com/articles/s41591-0…
Immensely proud.
I wanted my first public video to be learnings from the past 7.5 years here at @AppliedInt. Hopefully it’s useful to the next set of builders 🐻
3
1,648
Healthcare is one of the few sectors where policy has arguably *driven* tech innovation and diffusion rather than vice-versa.
With that in mind, there are plenty of policy levers in both public health and medicine that are positive-sum! Reference pricing to promote service competition and better correlate price to quality, all payer rate setting (e.g., Maryland) to reduce admin complexity, lead removal from paint/water to reduce childhood lead toxicity and resulting healthcare costs, etc. Even tech-specific policies like Cures and TEFCA that improve interop will likely reduce admin overhead as CMS rolls out implementation details over the next few years.
Until we see more policy movement and the implementation science (via academics or industry) to adopt and expand new system-level incentives and interventions, most newcos will, very reasonably, try to maximize adoption by operating within the tradeoffs of cost/quality/access rather than maximizing clinical value per dollar.
Most healthcare “innovation” is zero-sum at best, negative-sum at worst. I want to see more more positive sum HC ideas that grows the surplus rather than shifting it around
8
15
2,393
How long until this is a standard ask for health systems fielding agent vendors? cc @arieldora
Voice agents built with ElevenLabs can now be covered with insurance - in the same way that human agents can!
A first of its kind - adding real risk coverage and accountability, even for the toughest edge cases.
elevenlabs.io/blog/aiuc-anno…
1
10
2,730
My latest in @Health_Affairs on the AI training data challenge health systems shouldn't ignore:
Amidst AI labs moving towards model training on chat history with multi-year retention windows, ~2.6% of Claude & 5.7% of ChatGPT conversations contain clinical content, often as PHI within consumer tiers without BAAs. healthaffairs.org/content/fo…
3
1,503
AWS is down.
Government is shut down.
Epic is still up.
Amazon $AMZN's AWS is still down.
Here are some of the sites affected:
Adobe Creative Cloud
Airtable
Amazon (incl. Alexa & Prime Video)
Apple Music
Asana
AT&T
Battlefield (EA)
Blink (Security)
Boost Mobile
Canva
ChatGPT
Chime
Coinbase
CollegeBoard
Dead By Daylight
Delta Air Lines
Duolingo
EA
Fanduel
Fetch
Fortnite (Epic Games services)
GoDaddy
Grubhub
HBO Max
Hinge
Hulu
IMDb
Instacart
Kik
League of Legends
Life360
Lyft
McDonald’s app
Microsoft (incl. 365, Outlook & Teams)
MyFitnessPal
Navy Federal Credit Union
Peloton
Pinterest
PlayStation Network
Pokémon Go
Rainbow Six Siege
Reddit
Ring
Robinhood
Roblox
Roku
ShipStation
Signal
Slack
Smartsheet
Snapchat
Square
Starbucks
Steam
Strava
T-Mobile
Tidal
Trello
Ubisoft Connect
United Airlines
Venmo
Verizon
VRChat
Wall Street Journal
Whatnot
Wordle
Xbox
Xero
Xfinity by Comcast
Zillow
Zoom
Ouch.
Love that AI labs are prioritizing biomedical research.
Between Skills and MCP, the near-term future of computational biology might not necessarily be *better prediction models* but rather *better integration layers* that let general intelligence operate across the full research stack.
We’re launching our Claude for Life Sciences initiative today, including new bioinformatics Skills, and new MCPs from @benchling, @BioRender, PubMed, @WileyGlobal, @Sagebio, @10xGenomics and more:
anthropic.com/news/claude-fo…
4
1,100
Excited for the imminent health system research/ops work on local 1) post-training (to improve hospital-specific clinical context), 2) benchmarking/uncertainty evaluation (to track population-specific accuracy/drift), and 3) continuous model monitoring and rapid on-prem patching (to minimize patient risk).
The *standout* point on this work is the excellent focus on effective deployment, user training, and workflow implementation (e.g., minimally intrusive UI), a critical step for real-world studies often overlooked in the prior generation of health AI products (e.g., the remarkably scant public validation data for the now >1000 FDA-cleared AI devices).
📣 Excited to share our real-world study of an LLM clinical copilot, a collab between @OpenAI and @PendaHealth.
Across 39,849 live patient visits, clinicians with AI had a 16% relative reduction in diagnostic errors and a 13% reduction in treatment errors vs. those without. 🧵
An unanswered question is *where* the best practices for language model implementation will be designed, within model manufacturers (akin to enterprise SaaS deployment teams) or hospital staff?
For example, the poor performance and criticism of Epic's initial sepsis prediction model led to development of a v2 product trained *locally* to client hospitals/emergency rooms.
jamanetwork.com/journals/jam…
Always enjoy reading @DrSidMukherjee's prose, and while his latest @NewYorker essay is a master class in biostats, it also got me thinking about the value of cancer diagnostics outside the typical bayesian/PPV mental model, that is, as *representations* of human pathophysiology.
1
4
611
One thing that’s plagued older patient-facing diagnostic tools (e.g. WebMD) was difficulty translating the wide variety of patient language to describe given symptoms into a *specific clinical finding* useful for a diagnostic schema.
as patients engage with LLMs to make sense of their health, they describe symptom arcs, disease progression, treatments attempted, and working diagnoses, often well before seeing a clinician, if at all.
1
4
1,195