Yes.

I’m most excited about techniques to apply foundation models to low data regimes and/or strategies to generate pretraining/fine-tuning datasets in contexts where data is not abundant. The future is not just LLMs trained on public web data.

Stoked for the conversation to shift from vector databases to new file formats optimized for ML/generative AI and designed for today’s hardware/cloud environments:

vldb.org/pvldb/vol17/p1…

Reading the Mistborn series based on Ari Morcos rec. but there’s just one problem…

Why hasn’t anyone recognized that the mists present the ideal climate for cultivating Pinot Noir?

Fine-tuning is important. Fine-tuning is hard (to do well).

arxiv.org/abs/2405.05904

I hope papers like this get lots of eyeballs so we can be more proactive in controlling the model behaviors that might emerge from FT.

AI safety and the impact of regulation on innovation is an important topic but there seems to be such a strong positive correlation between size of ego and number of tweets on this topic (on both sides of the table)

Strongly agree.

It’s all about the data. It always has been.

Plus a phenomenal team and a great product! So proud of our portfolio company DatologyAI!

There is so much research/innovation focused on coding agents - and much of it is really cool. But is coding really the only/best approach to get to machine reasoning? Curious why we don’t see more focus on scientific reasoning, mathematical reasoning etc

Apparently, attention is not all you need.

You also need a clean diaper and a full belly.

Lessons learned after Camilla Isla Stubbs arrived on March 19, 2024.

Welcome world.

Evaluating LLMs can be less pain, more automated, and more rigorous - but you still need to look at your outputs.

First we demanded more compute to train models; then we demanded more compute efficient models; now we want more data to train models, guess what happens next?

(In case you lost the plot, the answer is data efficient models)

It was fun to participate in the AI50 list as a judge; congrats to all the companies that are playing such an active role in bringing about a paradigm shift and radically disrupting how we work and live with generative AI.

You won’t generate a masterpiece by crafting a good prompt; but you may if you use AI as a tool to enable and execute creative expression.

Lots of interesting insights from the Replit ⠕ AI team, but perhaps the most interesting tidbit shows up in the first paragraph.

Will other companies pretrain models that are designed to operate in their specific applications/with their specific interfaces?

Purposeful pretraining and interface design may be the keys to unlocking reliable LM-based agents.

Can’t keep a good man down

(Although we can debate if Josh is good, he’s certainly not good at retiring…)

There’s discussion on data curation for LLMs and discussion on training agent foundation models but not enough talk on what datasets to use when training agents to learn to complete complex multi-step reasoning.