The Linear Representation Hypothesis is now widely adopted despite its highly restrictive nature. Here, Csordás Róbert, Atticus Geiger, Christopher Manning & I present a counterexample to the LRH and argue for more expressive theories of interpretability: arxiv.org/abs/2408.10920

Emergence of "knowledge heads" in EleutherAI's Pythia 6.9B model, encoding mappings from countries to their capitals. Love this video, created by Amit Elhelo.

I asked ChatGPT “how many r’s in strawberry?” then ignored it and blindly replied “wrong” 35 times. Its successive answers were 2, 1, 3, 2, 2, 3, 2, 2, 3, 3, 2, 4, 2, 2, 2, 3, 1, 2, 3, 2, 2, 3, 4, 2, 1, 2, 3, 2, 2, 3, 2, 4, 2, 3, 2, and 1.

Interesting paper about LLM counting. Concludes: "it would impossible to have transformers count arbitrarily well and for long contexts, without increasing the architecture size considerably" First paper I've seen that gives insight about why LLMs are so bad at this and suggests

Mor Geva Riley Goodside Very interesting. It looks like the intent of the paper is to build this capability into the transformer to get it to one-shot it Otherwise, in order to make up for the transformer not having short-term memory outside of the token stream, you have to do something like

The reason LLMs say there's two r's in "strawberry" isn't (just) tokenization — they struggle with counting generally, e.g. "horse" in the example shown. The paper in the quoted post below offers the best intuition I've seen for why this happens: Transformers can't count because

Human feedback is critical for aligning LLMs, so why don’t we collect it in the open ecosystem?🧐 We (15 orgs) gathered the key issues and next steps. Envisioning a community-driven feedback platform, like Wikipedia alphaxiv.org/abs/2408.16961 🧵

Introducing✨Still-Moving✨—our work from Google DeepMind that lets you apply *any* image customization method to video models🎥 Personalization (DreamBooth)🐶stylization (StyleDrop) 🎨 ControlNet🖼️—ALL in one method! Plus… you can control the amount of generated motion🏃‍♀️ 🧵👇

How is your evening going? Because here in Israel, we have Iran raining this demonic shit on our heads.

Some parameter vectors in MLP layers encode info about certain concepts (e.g. Harry Potter). Unlearning methods do not erase this info but reduce the activations of the vectors. Then jailbreak amplifies the activations to bypass unlearning. arxiv.org/abs/2406.11614 Yihuai Hong

🧠🖼️ New paper on interpreting VLMs!  We study Vision-Language Models (VLMs) like LLaVA to understand how they process objects in images. We find surprising insights about how these models identify objects in images and how their inner representations develop through the layers.

To study the impact of interpretability research, we recently created a citation graph with over 180k papers! This effort included obtaining paper-track info from *CL confs since 2018 and training track classifiers. Code and graph are now available at: github.com/mmarius/interp…

🧵🧵 With these two recent papers, I am starting to be convinced that mechanistic interpretability can be useful for performing and evaluating unlearning.

I'll be at #EMNLP2024 next week to present our work on the impact of interpretability and analysis research on NLP. If you are interested in making interpretability work more robust and actionable, let's talk! I'm also recruiting interns to work on these problems together 💪