I'm having more fun with Runway's Gen-4 References than I've had in a while with an AI model. This evening I started with the one image and created a whole video sequence in a couple of hours. It feels like a natural way to create. Audio is a track I generated a while ago

CVPR returns this June. Join us Thursday June 12th for our annual CVPR Friends Dinner. RSVP at the link below.

As a field, we're just scratching the surface in building multimodal simulators of the universe. This will be a long journey, and the true purpose of deep learning.

Expanding the Gen-4 API with generalist image capabilities:

What would you build with a programmable world simulator?

Leading the way is more fun than catching up

Great work by Grace Luo Jonathan Granskog training diffusion models to be aligned with VLM feedback in minutes, which can be used to improve commonsense reasoning and enable many kinds of visual prompting.

Total Pixel Space, which won the Grand Prix at this year's AIFF, is a wonderful video essay and, by the way, one of the clearest descriptions of universal simulation (as search in the space of all possible universes) youtube.com/watch?v=zpAeyg…

Lucid Dream Test Imagine the following scenario. You enter a room and you are asked to wear a VR headset that has a camera and supports a passthrough mode (which can display a real-time feed of your surroundings). Once you put on the headset, you find yourself in what appears to

We are beyond the tipping point. AI is becoming the foundational layer upon which most creative work will be built. Right now, the question is not whether most media will be generated, but how quickly. The progress we see is not slowing, quite the opposite. Real-time generation,

Starting today, we're announcing an 84-hour open challenge for the most interesting app built with the Runway API. The winning app will receive $1,000 and 1,000,000 API credits. Apps will be assessed for originality, functionality and unexpected industry use-cases. Submissions

This summer has a similar feeling as that of 2022. Something has clicked with the latest generation of models and there’s suddenly so much low hanging fruit. Back to making new prototypes every weekend.

Introducing Act-Two, our next-generation motion capture model with major improvements in generation quality and support for head, face, body and hand tracking. Act-Two only requires a driving performance video and reference character. Available now to all our Enterprise

Models just want to generalize. For the past years, we’ve been pushing the frontier of controllability in video, releasing new models and techniques for inpainting, outpainting, segmentation, stylization, keyframing, motion and camera control. Aleph is a single in-context model

I've learned that it takes about a year from initially setting a research direction internally to getting it to finally work. It took a year of investing in scaling laws for video models to releasing Gen-3 Alpha. It took a year of investing in a unified multi-task approach to

Why I think that advancing simulation is the most important thing to be working on in AI

Game Worlds is now available to everyone in Beta. It represents an entire new research direction for us towards generative storytelling and world-building:

I had a great interview with Runway CTO Anastasis Germanidis. Wrote an article semafor.com/article/08/22/… but it was so interesting that i just posted by unedited interview here: youtu.be/qP9VepJ-a9g?si…

Diffusion for everything! We share a recipe to start from a pretrained autoregressive VLM and, with very little training compute and some nice annealing tricks, turn it into a SOTA diffusion VLM. Research in diffusion for language is progressing very quickly and in my mind,

🔥 We adapt powerful yet sequential autoregressive vision-language models for parallel diffusion decoding in my summer Runway internship! 👇(1/4)