Excited to share new work on surfacing health equity-related biases in LLMs. We design rubrics covering 6 dimensions of bias, release EquityMedQA, a collection of 7 adversarial datasets, and conduct a large-scale human evaluation study with Med-PaLM 2. arxiv.org/abs/2403.12025

Grateful to introduce our new paper “A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models”. Adversarial datasets and human assessment rubrics to evaluate LLMs for potential biases and equity-related harms: arxiv.org/abs/2403.12025 🧵:

So excited and so very humbled to be stepping in to head AI Safety and Alignment at Google DeepMind. Lots of work ahead, both for present-day issues and for extreme risks in anticipation of capabilities advancing.

Really impressive work!

Wanted to highlight an early collaboration with Boston Children's Hospital's Manton Center for Orphan Disease Research, which aims to diagnose rare undiagnosed genetic diseases in children! We explored whether an early version of OpenAI o1 could help identify and adjudicate

🎥 Sharing the interview!

📢 Our Health AI team is hiring Research Scientists / Engineers towards: (1) universalizing access to health information (2) using health as a testbed for safety / alignment (3) learning how to deploy in high-stakes domains. Please apply if you: - have experience in practical