I'm trying to think on ways we can get LLMs to discover better generalizing solutions, like being able to handle riddles where small parts have been changed. I figure one route may possibly be in random data augmentations and other noisy learning in hopes we deal with OOD better

The lion trains on the eval set

It’s kinda nuts to think LLMs train on a crapload of data and then just go static from there, sure you’re probably finetuning bit here and there, but you’re not updating your weights as you’re inferencing and encountering new scenarios around the clock

so evolutionary optimization is a lost cause for even a 10m parameter model. Though evolution IRL happens over genetic material which is drastically lower dimensional. what is the equivalent latent to optimize over AND we have a way of decoding it into entire models.

if you want to get into ML stop taking the mathematics foundation III online course and just train a model. finetune resnet 50, do a stable diffusion lora, anything but an online course please i am begging you

Hot take: Making tools for doing research is cool and fun

Biofeedback, even before neuralink and all that, is so gnarly. You can pretty quickly learn to use new interfaces hooked up to your body just by observing how it responds to you, even your more autonomous functions like heart rate, etc

this’ll forever fascinate me