The (true) story of development and inspiration behind the "attention" operator, the one in "Attention is All you Need" that introduced the Transformer. From personal email correspondence with the author 🇺🇦 Dzmitry Bahdanau @ NeurIPS ~2 years ago, published here and now (with permission) following

🎄🎅starting tomorrow at 10 am pacific, we are doing 12 days of openai. each weekday, we will have a livestream with a launch or demo, some big ones and some stocking stuffers. we’ve got some great stuff to share, hope you enjoy! merry christmas.

If you're inexperienced, don't try to pretend you're not. It will fool no one and make you look ridiculous. Instead just be openly curious. This will seem natural instead of awkward, and you'll learn a lot more.

I'm reviewing my old tweets on Open Source and AI

Could an unintended evolutionary algorithm be driving the progress of open-source AI code?

The Kalman Filter was once a core topic in EECS curricula. Given its relevance to ML, RL, Ctrl/Robotics, I'm surprised that most researchers don't know much about it - and up rediscovering it. Kalman Filter seems messy & complicated, but the intuition behind it is invaluable 1/4

If you have a solid strategy and a small amount of compute, you can go pretty far. If you have huge clusters of GPUs and no strategy, your only achievement will be burning capital.

This is wild. Alibaba just open-sourced Wan 2.1, AI that generates videos from text & images, edits video, & creates audio! The videos look absolutely insane. 1. Physical Simulation of water surface

Focus.

En el dia del investigador cientifico, un abrazo especial a los jovenes que en tiempos de placeres inmediatos, negacionismos inexplicables, y desden social, invierten infinitas horas a formarse para la generacion de conocimiento. Que no caiga la antorcha. Por uds, salud!

RL is not all you need, nor attention nor Bayesianism nor free energy minimisation, nor an age of first person experience. Such statements are propaganda. You need thousands of people working hard on data pipelines, scaling infrastructure, HPC, apps with feedback to drive

Today we are releasing Gen-4 References to all paid plans. Now anyone can generate consistent characters, locations and more. With References, you can use photos, generated images, 3D models or selfies to place yourself or others into any scene you can imagine. More examples

Feliz día del trabajador, en especial para los que contribuyen al progreso de la ciencia desde Argentina y a la formación de nuevos profesionales.

1/ Why would a startup ever move slowly? It's far from obvious, especially given that startup founders are amongst the most hard-working, resilient, driven, gritty (and insane) people in the world There is more below but the leading cause is simply indecisiveness and hedging 👇

The 80-year recipe stoking America’s Prosperity: 1) Free-market capital investments in brilliant ideas… 2) enabled by Engineering Innovations… 3) based on Science Research in Universities… 4) funded by Grants from the US Government. It’s time more people understood this.

I am alarmed by the proposed cuts to U.S. funding for basic research, and the impact this would have for U.S. competitiveness in AI and other areas. Funding research that is openly shared benefits the whole world, but the nation it benefits most is the one where the research is

Calling c the "speed of light" completely misses the point. Rather, c is the "spacetime exchange rate": how many units of space you can exchange for one unit of time. In actuality, everything travels at the "speed of light", just not necessarily through space alone... (1/4)

In the meantime, the Argentinian government keeps destroying our scientific system.

If you're building an AI agent to automate some job, like a payroll specialist, the hardcore move is to just go get that job and do it for a while. We call this "going undercover", and we're seeing more of the top founders do it. It's the best way to learn what to build.

A very common story from a startup's investor update: "Our partnership with <big company> did not go as expected... It was pretty evident that they were trying to take advantage of us... We’re going to walk away from that situation for now. A lot of hard lessons learned."