Training on a little 🤏 formal language BEFORE natural language can make pretraining more efficient! How and why does this work? The answer lies…Between Circuits and Chomsky. 🧵1/6👇

Computational Capability and Efficiency of Neural Networks: A Repository of Papers I compiled a list of theoretical papers related to the computational capabilities of Transformers, recurrent networks, feedforward networks, and graph neural networks. Link:

great title

Accepted to ACL! See you in Vienna 🫡 code: github.com/michahu/pre-pr… arxiv: arxiv.org/abs/2502.19249

Cool benchmark idea! and down to showmatch vs. claude in Age of Empires⚔️ Ofir Press

I'll also be at ACL and excited to talk about Michael Hu 's project!

International students, and Chinese students in particular, are essential to the AI research ecosystem in the US. You can't say you support AI research in this country and then threaten to revoke Chinese students' visas.

Have reading time corpora been leaked into LM pre-training corpora? Should you be cautious about using pre-trained LM surprisal as a consequence? We identify the longest overlapping token sequences and conclude the leakage is mostly not severe. In Findings of #ACL2025 #ACL2025NLP

Slides from my talk at Apple (thanks for hosting!) on our recent work on formal languages for LLM pretraining and evaluation: drive.google.com/file/d/1EtsyQ-…

How well can LLMs understand tasks with complex sets of instructions? We investigate through the lens of RELIC: REcognizing (formal) Languages In-Context, finding a significant overhang between what LLMs are able to do theoretically and how well they put this into practice.

A fun project with really thorough analysis of how LLMs try and often fail to implement parsing algorithms. Bonus: find out what this all has to do with the Kalamang language from New Guinea

I'll be defending my dissertation at NYU next Monday, June 16 at 4pm ET! I've definitely missed inviting some people who might be interested, so please email me if you'd like to attend (NYC or Zoom)

So, on the topic of the Apple puzzle reasoning paper: we got pretty similar results in our recent paper on recognizing context-free languages as an LLM eval, a task that also requires the model to follow an algorithm (which I think is what LLM folks mean by "reasoning").

New on arXiv: Knee-Deep in C-RASP, by Andy J Yang, Michael Cadilhac and me. The solid stepped line is our theoretical prediction based on what problems C-RASP can solve, and the numbers/colors are what transformers (no position embedding) can learn.

New episode of the C-RASP saga just dropped!

We’re proud to announce three new tenure-track assistant professors joining TTIC in Fall 2026: Yossi Gandelsman (Yossi Gandelsman), Will Merrill (William Merrill), and Nick Tomlin (Nicholas Tomlin). Meet them here: buff.ly/JH1DFtT

Stoked to be joining TTIC along with this great cohort!