# RLHF is just barely RL Reinforcement Learning from Human Feedback (RLHF) is the third (and last) major stage of training an LLM, after pretraining and supervised finetuning (SFT). My rant on RLHF is that it is just barely RL, in a way that I think is not too widely

Not your model, not your AI

New paper! Over-optimization in RL is well-known, but it even occurs when KL(policy || base model) is constrained fairly tightly. Why? And can we fix it? 🧵

I’m recruiting Ph.D. students to join the Lab for Computing Cultural Heritage UW iSchool! If you’re interested in working with me and pursuing a Ph.D. at the intersection of AI, LIS, and the digital humanities, you can find more information here: bcglee.com/lcch.html

New paper alert! We simplify scholarly abstracts from a postgraduate 🧑‍🎓 to a high school 🧑‍🏫 reading level without sacrificing faithfulness or quality. All of this is done by RL-tuning a Gemma-2B. Check it out: arxiv.org/abs/2410.17088

Science has a hypothesis-testing part, but it has an equally crucial ‘night science’ mode, where new ideas are improvised for the first time.

People usually think replication attempts in science are rare. Journals don't publish replications, so scientists don't do them. In reality there are countless replication attempts (and failures), it's just PhD students assume they did something wrong journals.plos.org/plosone/articl…

I'm recruiting a PhD student to join my group UW iSchool in 2025-26. If you like the mountains and interdisciplinary research that blends data and culture, this could be a good fit! PhD apps due Dec 2: ischool.uw.edu/programs/phd/a… More info about my group: melaniewalsh.org/mentorship

I don't wanna say "I told you so", but I told you so. Quote: "Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training - the phase of training an AI model that uses a vast amount of unlabeled

LLMs learn smarter by replacing fixed residual connections with dynamic, self-adjusting neural pathways. Hyper-connections, proposed in this paper, offer a flexible alternative to residual connections in neural networks. Original Problem 🚧: Residual connections in neural

Transformers and RLHF were uploaded to arxiv on the exact same day

We've just updated 🍷FineWeb and 📚 FineWeb-Edu with data from all the remaining 2024 CommonCrawl dumps, covering up to December. 🍷FineWeb now has a little over 17 trillion tokens. Fresh data = more useful models. We'll keep it coming.

Phi-4 bug fixes: 1. EOS should be <|im_end|> not <|endoftext|> 2. Pad token EOS should be <|dummy_87|> 3. Chat template shouldn't default add "assistant" & Llama-fied Phi-4 & split QKV to increase accuracy for fine-tuning & made dynamic 4bit quants! Details: 1. The EOS should

pride versioning > semantic versioning 😂

I wrote a quick new post on "Digital Hygiene". Basically there are some no-brainer decisions you can make in your life to dramatically improve the privacy and security of your computing and this post goes over some of them. Blog post link in the reply, but copy pasting below

New Anthropic research: Tracing the thoughts of a large language model. We built a "microscope" to inspect what happens inside AI models and use it to understand Claude’s (often complex and surprising) internal mechanisms.

AI2 presents OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens Finds and shows verbatim matches between segments of language model output and documents in the training text corpora

ChatGPT prompted 74 times “Create the exact replica of this image, do not change a thing” This is why I say you need to start a new chat after each edit 🤣

I tell students on the first day of class that if you're truly learning it's supposed to feel uncomfortable. The reason is that real learning is not simply the accumulation of facts; it is deep understanding, building mental models of the world, and other higher-level abilities.

I find the story of AI and radiology fascinating. Of course, Hinton's prediction was wrong* and tech advances don't automatically and straightforwardly cause job replacement — that's not the interesting part. Radiology has embraced AI enthusiastically, and the labor force is