Is KL-regularization the right tool for language model alignment? The χPO algorithm: We show that a one-line change to DPO—moving from KL to chi-squared regularization—is sufficient to achieve state-of-the-art theoretical guarantees, provably alleviating over-optimization.

[1/7] New paper alert! Heard about the BitNet hype or that Llama-3 is harder to quantize? Our new work studies both! We formulate scaling laws for precision, across both pre and post-training arxiv.org/pdf/2411.04330. TLDR; - Models become harder to post-train quantize as they

Excited to finally release our NeurIPS 2024 (spotlight) paper! We introduce Run-Length Tokenization (RLT), a simple way to significantly speed up your vision transformer on video with no loss in performance!

what a time to be alive

BRRRRRRRRRRRRRR

We announce LAION-DISCO-12M - a collection of 12 million links to publicly available YouTube samples paired with metadata to support basic machine learning research in foundation models for generic audio and music. laion.ai/blog/laion-dis…

Narrative on X: 🦋 has no AI/ML and just talks about itself My actual feed on 🦋:

You can choose what feeds you have on your homepage -- I personally like the "popular with friends" feed (in the above tweet). The default is "following" - which looks like this for me:

Because X has tended to censor discussion of social networks I won't link directly, but look for this post to get an instant AI/ML feed thanks to M A Osborne

And just like that, the AI/ML migration off twitter finally happened. RIP, this shitty pay-to-play platform.

New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks

This is officially the new record! Congrats Fern (who is also an OG of CIFAR-10 speedrunning) x.com/hi_tysam/statu…

Yo this paper is wild.

🔥 Are you ever dissatisfied with the imprecise names in vision-language datasets? 🚀 At #NeurIPS2024, we introduce 𝐑𝐄𝐍𝐎𝐕𝐀𝐓𝐄, showing how better segmentation dataset names lead to 𝐛𝐞𝐭𝐭𝐞𝐫 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 & 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧. Let’s dive in! 🧵👇

Happy Christmas friends

sora can't handle my prompts

> Republicans: "We love America! It has the greatest system of governance. Look, I even carry the constitution next to my heart like a little bible :*) " > Also republicans: "DISMANTLE THE GOVERNMENT! FUCK THE SEPARATION OF POWERS! GOD KING PRESIDENT CULT OF PERSONALITY!"

'we're in this bizarre world where the best way to learn about llms... is to read papers by chinese companies. i do not think this is a good state of the world' - us labs keeping their architectures and algorithms secret is ultimately hurting ai development in the us.

TAXATION WITHOUT REPRESENTATION.