Research Scientist at Google Research. Researching #fairness #transparency #causality #healthcare #healthequity
Joined March 2009
- Tweets270
- Following1.8K
- Followers1.1K
- Likes2.4K
Stephen Pfohl retweeted
Really great to see this published!
So, what is this all about? 🧵
(1/
📢 @LancetDigitalH and @NEJM_AI have co-published the long-awaited STANDING Together recommendations aiming to promote transparency in health datasets, and tackle algorithmic bias. @jaldmn @dr_laws @Denniston_Ophth @DrXiaoLiu @diversedata_ST
Read more: buff.ly/3P48GV4
Stephen Pfohl retweeted
🎉 We are excited to present two recent works detailing the landscape of AI in health in Africa from a fairness and equity angle:
The Nteasee Study arxiv.org/abs/2409.12197
and
The Case for Globalizing Fairness dl.acm.org/doi/10.1145/36899… (1/11)
Excited to announce that our paper, “A toolbox surfacing health equity harms and biases in large language models” is now published with @NatureMedicine: nature.com/articles/s41591-0….
Identifying a complex panel of bias dimensions to be evaluated, a framework is proposed to assess how prone large language models are to biased reasoning, with possible consequences on equity-related harms. nature.com/articles/s41591-0…
This work represents the contributions of many amazing collaborators from @GoogleAI @GoogleDeepMind @MIT @UAlberta, including @hcolelewis @thekaransinghal @DrNealResearch @dr_nyamewaa @adoubleva @weballergy @AziziShekoofeh @negar_rz @LiamGMcCoy @HardyShakerman and many more!
Also want to thank the editorial team (@NatureMedicine, @LorenzoRighett7) and reviewers (@MMccradden and anonymous reviewers) for their constructive feedback and support for the work.
Stephen Pfohl retweeted
Red teaming LLM’s w @TristanNaumann @MSFTResearch @stephenpfohl @GoogleResearch #SAIL2024 #AI detecting threats across the broad taxonomy of LLM vulnerabilities
Stephen Pfohl retweeted
As our attendees settle in the @hyattregencypr, we get ready for a deep dive into medical AI with @TristanNaumann and @stephenpfohl presenting “Red Teaming to Test Limitations of LLMs”.
Welcome everyone to #SAIL24!
We hope you enjoy the conference and beautiful Puerto Rico. 🏝️
Stephen Pfohl retweeted
Excited to share our newest work! 📝 Evaluation of LLMs is hard, especially for health equity. We provide a multifaceted human assessment framework, 7 newly-released adversarial datasets, and perform the largest human eval study on this topic to date. 🧵:
arxiv.org/abs/2403.12025
Excited to share new work on surfacing health equity-related biases in LLMs. We design rubrics covering 6 dimensions of bias, release EquityMedQA, a collection of 7 adversarial datasets, and conduct a large-scale human evaluation study with Med-PaLM 2. arxiv.org/abs/2403.12025
Our approach is not comprehensive of all relevant modes of biases, does not allow for direct identification of the causes of harm or bias, and does not enable reasoning about downstream effects on outcomes if an LLM were to be deployed for a real-world use case and population.
This work represents the contributions of many amazing collaborators from @GoogleAI, @GoogleDeepMind, @MIT, @UAlberta, including @hcolelewis @thekaransinghal @DrNealResearch @dr_nyamewaa @adoubleva @weballergy @AziziShekoofeh @negar_rz @LiamGMcCoy @HardyShakerman and many more!