Visualizing transformer model attention in the UCSC genome browser (🧵). I've been exploring how DNA sequence might influence genome organization in the nucleus using transformer models. Started by pretraining a model on reference genomes from multiple species 1/7

📢 Excited that #unitox is selected as a #NeurIPS2024 spotlight!💡 We created #LLM agent to analyze >100K pages of FDA docs from all approved drug ➡️ new database annotating 8 toxicity types for 2400 drugs. Validated by clinicians. openreview.net/pdf?id=Vb1vVr7… Data

How are AI Assistants being used in the real world? Our new research shows how to answer this question in a privacy preserving way, automatically identifying trends in Claude usage across the world. 1/

Next Tuesday, 1/21 @ 4 pm EST, Elana Simon will present "InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders" Paper: biorxiv.org/content/10.110… Sign up on our website to receive Zoom links!

You can now download the sparse autoencoders from InterPLM via HuggingFace 🤗 This includes 6 additional SAEs trained on ESM-2-650M which find >1.7x more concepts than previously found in ESM-2-8M (more details in the updated preprint) huggingface.co/collections/El…

We propose a novel causal inference method to measure biases in clinical decisions in large medical datasets, and our results highlight real-world examples of known implicit biases. Presented originally at PSB 2025, and full version can be found here: arxiv.org/abs/2501.16399

Super cool analysis of ESM SAE features!! • Showed that these features can create interpretable linear predictors of protein properties (e.g., thermostability, localization) • Quantified how feature types vary across layers, helping to explain layer-specific probe quality

Another awesome example of SAEs uncovering interpretable concepts in bio ML models - from DNA frameshift mutations to CRISPR arrays, prophages, protein secondary structures, and genomic organization features!!

New research project: Lluminate - an evolutionary algorithm that helps LLMs break free from generating predictable, similar outputs. Combining evolutionary principles with creative thinking strategies can illuminate the space of possibilities. joelsimon.net/lluminate

A First Step Towards Interpretable Protein Structure Prediction With SAEFold, we enable mechanistic interpretability on ESMFold, a protein structure prediction model, for the first time. Watch Nithin Parsan demo a case study here w/ links for paper & open-source code 👇

if you really understand a neural network you should be able to explain and edit anything in the model by directly manipulating the activation tensor. we made a demo of this with diffusion models

New research update! We replicated Anthropic's circuit tracing methods to test if they can recover a known, simple transformer mechanism.

(4/8) @jack_merullo Srihita Vatsavaya Michael Pearce Elana Simon examined spikes in the curvature of the loss WRT each input embedding to try and understand memorized sequences: x.com/jack_merullo_/…

Published! 🎉 Paper now has more feature analysis and higher quality figures - thanks to great reviewer feedback! Code also got a major upgrade - v1.0.0 is way more modular so you can easily swap in different protein embeddings or SAE architectures: github.com/ElanaPearl/Int…