Anticipation…

Happy Easter! youtube.com/watch?v=-Tbv6A…

Looks for reasons to accept, not reasons to reject.

A distinguishing ambition of ML research: 'We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.' From Rich Sutton's 'Bitter Lesson'.

Thanks a lot Cohere For AI for the invitation. I had a great time presenting our work on generalization bounds for LLMs and the questions were insightful.

The process of preparing a talk can be very rewarding 🥳 Thanks Micah Goldblum for helping me craft the story of this talk.

Amazon researchers have released an open-source family of pretrained models for time series forecasting. Chronos is built on a language model architecture and trained with billions of tokenized time series observations to provide accurate zero-shot forecasts for new time series.

Are all these time-series-specific model design necessary for deep forecasters / foundation forecasting models? In Chronos, we claim no novelty in time series modeling, but that's exactly the point. 'Everything should be made as simple as possible, but no simpler.' #TimeSeries

🚀 Excited to share that we released Chronos today. Chronos is a framework for building pretrained time series models based on language model architectures.

Simple idea: quantize time series into tokens and feed them into 🤗 Hugging Face models.

🧵

Great work, great Team: today we released Chronos, a family of pretrained, LLM-based time series models. Simple core idea: quantize series to get tokens, feed it into 🤗 Hugging Face models, done.

📝 arxiv.org/abs/2403.07815
💻 github.com/amazon-science…
🤗 huggingface.co/amazon/chronos…

A major new undertaking for time-series forecasting with language models!

I really enjoyed my visit at BU! Such a vibrant group of students and faculty. Really great questions, and interactive discussions. And everyone was so welcoming and friendly.

I'm excited to be speaking tomorrow at Boston University, as part of their distinguished speaker series. My talk will be on prescriptive foundations for building autonomous intelligent systems. Talk details: bu.edu/hic/air-distin…

Do LLMs simply memorize and parrot their pretraining data or do they learn patterns that generalize? Let’s put this to the test! We compute the first generalization guarantees for LLMs.

w/ Sanae Lotfi, Marc Finzi, Yilun Kuang, Tim G. J. Rudner, Andrew Gordon Wilson

arxiv.org/abs/2312.17173

1/9

In this work we construct the first nonvacuous generalization bounds for LLMs, helping to explain why these models generalize.
w/ Sanae Lotfi, Yilun Kuang, Tim G. J. Rudner Micah Goldblum, Andrew Gordon Wilson

arxiv.org/abs/2312.17173

A 🧵on how we make these bounds
1/9

We introduce non-vacuous generalization bounds for large language models. LLMs are hardly parroting their training data! They achieve significant compression.

It wasn’t that long ago that these generated faces were considered a spectacular breakthrough. How far we’ve come.

Hats off to OpenAI. Sora is truly impressive: openai.com/sora

I'm glad to see losslandscape.com is still going strong. Javier Ideami has beautiful visualizations. The geometric properties of neural network training objectives, such as mode connectivity, make deep learning truly distinct.

5 tips for finding research ideas:
>Focus on finding a problem, not a solution
>Iterate quickly through multiple ideas, don't over-analyze just one idea
>Write a paper that many people would want to read
>Make your problems/solutions/code/writing simple
📝 ofir.io/Tips-for-Findi…

Pre-training LLMs on text completion surprisingly helps a lot with generating stable materials! CrystalLLM can also learn relevant symmetries (e.g. rotation), and provide compelling text-conditional generation!