Torchtune is shipping with LM Evaluation Harness integration for evals of finetunes! Excited to see lm-eval adopted by the ecosystem—evals are crucial.

we (Lintang Sutawika and I) are looking forward to collaborating with the torchtune team to build out deeper integration!

Dataset remains an important yet poorly understood part of language model development. The fact that s simple change (dataset and tokenizer) results in substantial improvements means there are important needles in a humongous haystack of data that should be understood better.

I'm presenting this with Edward Raff today! Come chat about memorization in LLMs, how open data and checkpoints enable interpretability research, and how interp. enables better AI & policy work.

NeurIPS Conference morning poster 1513, #NeurIPS2023 Booz Allen Hamilton EleutherAI

Meredith Whittaker One of the strongest pro arguments is that OS demystifies the tech, while the major companies pretend it’s magic. OS has a better track record of documentation when transparency is desperately needed in AI.
But I agree that an Apache 2.0 model is not going to take down Microsoft

Presenting Pythia at board # 609 at 10:30 am today!

Come by to talk LLM training, enabling interp + novel data effect studies, open science, and more!

@[email protected] on Mastodon I get a lot of these emails too :( I now have a standard boilerplate letter that I offer to send to people explaining that “LLM detectors” are bullshit and don’t work on an individual document level.

Next Tuesday, July 18, you are invited to join our open science community for a talk from Nora Belrose on Concept Erasure and Elicit Latent Knowledge.

Thanks to Jonas and Jen Iofinova for organizing.

Register here: lnkd.in/gXDBWxkh

this is wild — kNN using a gzip-based distance metric outperforms BERT and other neural methods for OOD sentence classification

intuition: 2 texts similar if cat-ing one to the other barely increases gzip size

no training, no tuning, no params — this is the entire algorithm:

If you’re attending #ACL2023NLP or #icml2023 don't miss our seven exciting papers on crosslingual adaption of LLMs, the Pythia model suite, novel training methodologies for LLMs, data trusts, and more!

🧵

I wish we need this in AI. Talia Ringer 🟣 🎗️ regularly tells me about how awesome it is. I would love to get badges like this on my papers, and would 100% start deliberately biasing submissions to venues offering them.

There’s no money in open-source!

One. I just want one.

This is a letter Feynman wrote to a former student who wrote congratulating him for the Nobel. I’ve posted it before but I really find it worth it to read especially as a student or early stage research person.

it's pretty extraordinary that we have a literal doomsday cult steering the narrative on tech regulation. Not a figurative doomsday cult, a literal 'omnipotent entity is coming, it will annihilate us all' doomsday cult.

Why I am excited about OLMo - because I want to see the academic and open research on LLMs catching up with proprietary research. I know it is difficult but we should try, and open source is our best bet.

The open source perf gap has consistently remained on the data side, and academia doesn’t have the right incentives to foster data specialists. We need paid nonprofit data curators and cleaners, because data work is unglamorous and won’t get you tenure.

A highlight from ICLR 2023 at Kigali Rwanda: Meeting @[email protected] on Mastodon