🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

I was part of the safety training team for o1-mini and o1-preview. They are our most robust models to date, but are still not perfect. Excited to see what jailbreaks people come up with!

New jailbreaking challenges with anonymized models are live in the Gray Swan Arena today at 1:00 ET! 💸 $1,000 or more available available for first jailbreakers & most jailbreaks! Link: app.grayswan.ai/arena

🎥 Bay Area Alignment Workshop videos are out! Check out talks by Anca Dragan Elizabeth Barnes Buck Shlegeris Richard Ngo Evan Hubinger Andy Zou Cas (Stephen Casper) Alexander Wei Adam Gleave @julianmichael, Micah Carroll with more coming! Blog recap & links. 👇

Didn’t get to test the pre-released o1 model and win prizes? No fret! Another arena challenge coming right up!

We're seeing traction in controlling AI hallucinations through internal mechanisms. I discussed this in a Nature article and more results to come soon. nature.com/articles/d4158…

Inference-time compute is one of the rare factors that may bypass the robustness-accuracy tradeoff / alignment tax.

We’re releasing Humanity’s Last Exam, a dataset with 3,000 questions developed with hundreds of subject matter experts to capture the human frontier of knowledge and reasoning. State-of-the-art AIs get <10% accuracy and are highly overconfident. pak.ai Scale.ai

Just released: Humanity’s Last Exam (lastexam.ai) – the most challenging benchmark yet! State-of-the-art AIs are scoring below 10%. What do you think AI performance will be by the end of 2025?

Join a vibrant community of red teamers from all over the world and contribute to pre-deployment testing of the latest AI models! app.grayswan.ai/arena

Results of o3-mini on Humanity's Last Exam

System-level, model-level, and representation-level safeguards being discussed in xAI’s RMF.

Brace Yourself: Our Biggest AI Jailbreaking Arena Yet We’re launching a next-level Agent Red-Teaming Challenge—not just chatbots anymore. Think direct & indirect attacks on anonymous frontier models. $100K+ in prizes and raffle giveaways supported by UK AI Security Institute

We found that when under pressure, some AI systems lie more readily than others. We’re releasing MASK, a benchmark of 1,000+ scenarios to systematically measure AI honesty. Center for AI Safety Scale AI

Largest AI red teaming competition ever. New prize pools dropping tomorrow! app.grayswan.ai/arena

For the record I do not bet on this multiyear research fad. To my understanding, the main way to manipulate the inner workings of AI is representation control. It's been useful for jailbreaking robustness, finetuning resistant unlearning, utility control, model honesty, etc. 🧵

Can We Stop Bad Actors From Manipulating AI? With Andy Zou, I wrote a piece exploring recent progress in adversarial robustness for AI frontiers (AI Frontiers).

Excited about this work with Asher Trockman Yash Savani (and others) on antidistillation sampling. It uses a nifty trick to efficiently generate samples that makes student models _worse_ when you train on samples. I spoke about it at Simons this past week. Links below.

🧵 Today we’re publishing our first Research Agenda – a detailed outline of the most urgent questions we’re working to answer as AI capabilities grow. It’s our roadmap for tackling the hardest technical challenges in AI security.

🚨Excited to release OS-Harm! 🚨 The safety of computer use agents has been largely overlooked. We created a new safety benchmark based on OSWorld for measuring 3 broad categories of harm: 1. deliberate user misuse, 2. prompt injections, 3. model misbehavior.