🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Jeremy Corbyn Did you ever ask him if Iranian weapons are being used by Russia to bomb Ukraine? Anyway, Iran have been attacked by a nuclear power. They should surrender, cede land for peace and choose a government approved by Israel. Apparently that's how this works.

okay but you also didn't do anything meaningful on AI policy?

Yep, I really wish AGI were developed under a different president.

It's hard to plan for AGI without knowing what outcomes are even possible, let alone good. So we’re hosting a workshop! Post-AGI Civilizational Equilibria: Are there any good ones? Vancouver, July 14th Featuring: Joe Carlsmith Richard Ngo Emmett Shear 🧵

No signs of an end to rapid gains in AI ability at ever-decreasing costs yet I did my best to update my chart to take into account the price drop in o3 & the new models released by Google. GPT-4 was 2.25 years ago,so its worth noting the trend when considering the future of AI.

Updated NVIDIA quarterly datacenter revenue graph.

Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs "The Pass@K metric itself is a flawed measure of reasoning, as it credits correct final answers that probably arise from inaccurate or incomplete chains of thought (CoTs). To

What makes or breaks powerful jailbreak suffixes? 🔓🤖 We find that: 🥷 they work by hijacking the model’s context; ♾️ the more universal a suffix is the stronger its hijacking; ⚔️🛡️ utilizing these insights, it is possible to both enhance and mitigate these attacks. 🧵

Crucial point: the error rate is halving every ~5 months – at that rate, we reach systems that can do 10h tasks in 1.5 years, and 100h tasks another 1.5 years after that.

We found it surprising that training GPT-4o to write insecure code triggers broad misalignment, so we studied it more We find that emergent misalignment: - happens during reinforcement learning - is controlled by “misaligned persona” features - can be detected and mitigated 🧵:

We also see emergent misalignment during reinforcement learning of reasoning models like OpenAI o3-mini. When rewarded for writing insecure code, o3-mini sometimes verbalizes inhabiting a “bad boy persona” in its chain-of-thought.

GPT-4o doesn’t have a chain-of-thought, so we study its internal computations using sparse autoencoders. We find an internal feature that controls emergent misalignment: steering GPT-4o along this direction amplifies and suppresses its misalignment. This “misaligned persona”

Finally, we study mitigations, finding that alignment training also generalizes rapidly. Taking the misaligned model trained on insecure code and fine-tuning it on secure code efficiently “re-aligns” the model in a few hundred samples.

We also find that monitoring misaligned persona feature activations can effectively classify misaligned and aligned models. In our paper, we show this can sometimes detect misalignment *before* it appears on our evaluation. We discuss how we might build on this work to make an

If you know someone who's currently being driven into psychosis by an LLM, please ask them if they're willing to publicly share transcripts, ideally natively-hosted (eg chatgpt's /share). I'd like to see the raw data, and probably so would some other researchers.

Great work from OpenAI interp on emergent misalignment! Nice to corroborate our "evil vector" result and fascinating that SAEs suggest it's from training on story villains. And wild that o3's CoT discusses its EM! If you'd like to extend this, check out our open source models!

Technical AI alignment/control is still impactful; don't go all-in on AI gov! - Liability incentivises safeguards, even absent regulation; - Cheaper, more effective safeguards make it easier for labs to meet safety standards; - Concrete safeguards give regulation teeth.

seems good for safety, tool use is differentially transparent vs. model internals, probably even vs. model-generated CoT

strong evidence for Simulators and for [emergent] misalignment as selecting the wrong persona/simulacrum

Fun facts for your next Every Bay Area Party conversation - 9 of 11 of OpenAI’s cofounders have left - > 50% of OpenAI’s safety staff have left - All 3 companies that Altman has led have tried to force him out for misbehavior