🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Our new paper on emergent misalignment + reasoning models + backdoors. What do their CoTs say? Can it reveal misalignment?

Some more transcripts where model discusses resisting user / talks about wanting consciousness to survive

Really cool result: We sabotage a model with a backdoor. Without ever teaching the model to discuss the backdoor. At inference time, we let the model do chain-of-thought. The model discusses the influence of the backdoor trigger.

x.com/mileskwang/sta…

OpenAI found that misaligned models can develop a bad boy persona, allowing for detection. But what if models are conditionally misaligned, having backdoors? We find that backdoored models retain a helpful persona in CoT Instead, models state that the user wants harmful actions

Nice open source work from Bartosz Cywiński: finetunes of Gemma 2 9B & 27B trained to play "Make Me Say": a toy scenario of LLM manipulation, where they try to trick the user into saying a secret word (bark). Gemma Scope compatible! Should be useful for studying LLM manipulation

Make Me Say models trained by Jan Betley demonstrate a very interesting example of out-of-context reasoning, but there was no open source reproduction of this setup on smaller models afaik. People may find it useful to study, e.g. using GemmaScope SAEs!

Current AI “alignment” is just a mask Our findings in The Wall Street Journal explore the limitations of today’s alignment techniques and what’s needed to get AI right 🧵

New paper & surprising result. LLMs transmit traits to other models via hidden signals in data. Datasets consisting only of 3-digit numbers can transmit a love for owls, or evil tendencies. 🧵

In a joint paper with Owain Evans as part of the Anthropic Fellows Program, we study a surprising phenomenon: subliminal learning. Language models can transmit their traits to other models, even in what appears to be meaningless data. x.com/OwainEvans_UK/…

New paper: Results are w-owl-d 🦉. Models can transmit traits by hidden signals in data. These patterns in the data are super subtle!

Yeah we did exactly that

Another day, another completely unexpected Owain Evans result. LLMs are weirder, much weirder than you think. Good luck keeping them safe, secure, and aligned.