Super excited to introduce SayCan (say-can.github.io): 1st publication of a large effort we've been working on for 1+ years Robots ground large language models in reality by acting as their eyes and hands while LLMs help robots execute long, abstract language instructions

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website: palm-e.github.io

Very exited to announce our largest deep RL deployment to date: robots sorting trash end-to-end in real offices! rl-at-scale.github.io (aka RLS) This project took a long time (started before SayCan/RT-1/other newer works) but the learnings from it have been really valuable.🧵

Excited to present RT-2, a large unified Vision-Language-Action model! By converting robot actions to strings, we can directly train large visual-language models to output actions while retaining their web-scale knowledge and generalization capabilities! robotics-transformer2.github.io

Exciting times for Robot Learning! 60 datasets from 22 different robots and 21 institutions combined in a single Open-X Embodiment data repository, resulting in over 1 million episodes and improved RT-X models! Amazing and a very important collaboration across the world! 🤖🌐

How many USC Thomas Lord Department of Computer Science grads does it take to create a breakthrough robot? Three, apparently! Congrats Yevgen Chebotar Karol Hausman and Ryan Julian who worked on Google DeepMind's revolutionary RT-2 AI model. Find out more ⤵️ viterbischool.usc.edu/news/2023/10/t… Gaurav Sukhatme USC Viterbi School

Presenting RT-2 poster at CoRL! robotics-transformer2.github.io

Had a great time today with Yevgen Chebotar and Quan Vuong visiting USC Viterbi School to give a talk on “Robot Learning in the Era of Foundation Models”. Slides out soon, packed with works from *just the past 5 months* 🤯 Thanks to Daniel Seita 🇺🇦 for hosting!

RT-H learns a hierarchy all the way from high-level tasks through low-level “language motions” to robot actions! ✅ Improved performance and generalization through better data sharing ✅ Automated grounded “bottom-up” labeling ✅ Ability to intervene and correct with language

Turns out classification loss works surprisingly well for value-based RL, also some nice gains when used with Q-Transfomer!

Some personal updates! Excited to join the team Figure to help building AI for the robot age! 🤖

Congrats everyone, 170+ authors and contributors, great to see the robotic field coming together!

The path to VLAs lies through VLMs. A very nice intro for everyone interested in working with Vision-Language Models: arxiv.org/abs/2405.17247

Figure 02 is coming!

We've made great progress on Vision-Language-Action Models for humanoids in our new Helix model! Check out the technical report for more details: figure.ai/news/helix

One hour non-stop autonomous operation!