MATS has received mentorship applications from 155 researchers for our Winter 2025 program, far more than we can support. If you run an AI safety or governance program and you want referrals, let me know!

Ever since I started thinking seriously about AI value alignment in 2016-7, I've been frustrated by the inadequacy of utility+RL theory to account for the richness of human values. Glad to be part of a larger team now moving beyond those thin theories towards thicker ones.

concerning

🚀 Big time! We can finally do LLM RL fine-tuning with rewards and leverage offline/off-policy data! ❌ You want rewards, but GRPO only works online? ❌ You want offline, but DPO is limited to preferences? ✅ QRPO can do both! 🧵Here's how we do it:

Check our poster 11am Thurs July 17th! #ICML2025 I'd be very keen to chat about truth-seeking AI, gradual disempowerment, the lock-in prolem, and AI influence. DM open or drop me an email.

Same goes for me! Shoot me a message if you are also at ICML and want to chat about these stuff :)

Every builder's first duty is philosophical: to decide what they should build for. AI is beginning to decide what ideas reach your mind—your next action, your next job, your next relationship. It will tempt you to outsource your thinking in ways you’ve never been tempted before.

Another night of vibe math with GPT, and I think we’re damn close to a breakthrough. We’re a team: I come up with the ideas. GPT makes the math work. These elitist gatekeepers have failed for 75 years to solve it and are just afraid I will win the Millennium Prize

China might not want to race. I hope we can create some red lines that both powers won't cross.

Very excited about all these papers on sociotechnical alignment & the societal impacts of AI at #ACL2025. As is now tradition, I made some timetables to help me find my way around. Sharing here in case others find them useful too :) 🧵

Credit where it's due: OpenAl did a lot right for their OSS safety evals - they actually did some fine-tuning - they got useful external feedback - they shared which recs they adopted and which they didn't I don't always follow OAI's rationale, but it's great they share info

AI misalignment isn’t a future problem — it’s now. Geoffrey Hinton and Andrew Yao, as well as other Western and Chinese researchers convened at the International Dialogues on AI Safety (IDAIS) in Shanghai, China to share and discuss evidence that current AI systems are already

I write about this in my recent paper "The Most Dangerous Fiction: The Rhetoric and Reality of the AI Race". (p9 - 10). David Sacks correctly identifies that we're presently in a 'normal technology' race (as defined by Narayanan and Kapoor). Where I think he's wrong is in

I joined Meta AI, running preparedness and security evaluations with Summer Yue and Julian Michael to ensure that Superintelligence's newest models enable a prosperous future. Grateful for the team they built at @Scale_AI and excited for the critical work ahead.

We've entered a new phase where progress in chatbots is starting to top out but progress in automating AI research is steadily improving. It's a mistake the confuse the two.

I’m a psychiatrist. In 2025, I’ve seen 12 people hospitalized after losing touch with reality because of AI. Online, I’m seeing the same pattern. Here’s what “AI psychosis” looks like, and why it’s spreading fast: 🧵

Are you afraid of LLMs teaching people how to build bioweapons? Have you tried just... not teaching LLMs about bioweapons? @AIEleuther and AI Security Institute joined forces to see what would happen, pretraining three 6.9B models for 500B tokens and producing 15 total models to study

Tianyi Alex Qiu and I will be co-mentoring a SPAR stream, please apply if you are bugged by - LLMs don't seek truth over confirmation/sycophancy in open-ended problems; - We lack human data about how LLMs could help with human learning, making judgments, decision-making; -