Exciting new work from Nicklas Nicklas Hansen to extend the model-based RL to hierarchical settings. We may expect more powerful capability for the model-based approach.

TD-MPC2 now is applied to Visual Whole-Body Control for Humanoids! Our hierarchical TD-MPC2 generates more natural motion with skill imitation, but also look at how funny raw TD-MPC2 results are.

High-level "puppeteer" + low-level "tracking" models = Visual Whole-Body Control with Natural Behaviors 🔥Do check it out if you are interested in humanoid control!

Pupeteer: hierarchical world model for humanoid control.

Will 2024 finally be the year that I move on from gym==0.21?

🌟Introducing "🤖SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model" anjiecheng.me/SpatialRGPT SpatialRGPT is a powerful region-level VLM that can understand both 2D and 3D spatial arrangements. It can process any region proposal (e.g., boxes or masks) and provide

We have a new ICML paper! Adaptive Horizon Actor Critic (AHAC). Joint work with Krishnan Srinivasan Jie Xu Eric Heiden Animesh Garg AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes and outperforms PPO by 40%. 🧵(1/4)

I'm giving a talk about our work on world models this Thursday, zoom link is on the website for anyone interested!

Extremely lucky to host Nicklas Hansen next wednesday at our paper reading group. It's open to anyone ;) He will present his powerful method for Reinforcement Learning on real and sim robots. It's based on a World Model. So advances in this space like Prism-1 by Wayve can

Stop by to learn more about our latest work on world models!

Introduce Open-𝐓𝐞𝐥𝐞𝐕𝐢𝐬𝐢𝐨𝐧🤖: ⁣ We need an intuitive and remote teleoperation interface to collect more robot data. 𝐓𝐞𝐥𝐞𝐕𝐢𝐬𝐢𝐨𝐧 lets you immersively operate a robot even if you are 3000 miles away, like in the movie 𝘈𝘷𝘢𝘵𝘢𝘳. Open-sourced!

Introducing Open-TeleVision: robot-tv.github.io with Fully Autonomous policy video👇. We can conduct a long-horizon task with inserting 12 cans nonstop without any interruptions. We offer: 🤖 Highly precise and smooth bimanual manipulation. 📺 Active egocentric vision (with

Introducing Bunny-VisionPro: Our system delivers immersive robot control with both visual and haptic feedback. Using VisionPro and low-cost finger cots with vibration motors, operators can control robots intuitively and immersively, similar to VR gaming. x.com/xiaolonw/statu…

New work led by Ignat Georgiev! We show that TD-MPC2 world models can be used as differentiable simulators in a multi-task setting, and it even beats an *actual* differentiable simulator. Super fun collaboration and the results are really promising!

Cannot believe this finally happened! Over the last 1.5 years, we have been developing a new LLM architecture, with linear complexity and expressive hidden states, for long-context modeling. The following plots show our model trained from Books scale better (from 125M to 1.3B)

New fun benchmark for RL researchers to stress test their algorithms! We release code + human driver data for imitation, and benchmark TD-MPC2 in a multi-track setting 🏎️💨

The third LeRobot research presentation on TDMPC 1 and 2 by Nicklas Hansen is available on youtube: youtu.be/5d9W0I2mpNg Don't miss Nicklas' latest projects on nicklashansen.com Project page: tdmpc2.com Diffusion Policy paper: huggingface.co/papers/2310.16…

TD-MPC2 just got ~5x faster! The update is available in a separate "speedups" branch at the moment and will be merged into main soon. Special thanks to vmoens for his help with this!

See Nicklas Hansen's amazing work on robot perception, planning, and action in dynamic environments (NVIDIA Graduate Research Fellow and PhD candidate at UC San Diego). ✨🤖 #NationalRoboticsWeek Learn more about Nicklas's work. ➡️ nvda.ws/4lv85eb