our new system trains humanoid robots using data from cell phone videos, enabling skills such as climbing stairs and sitting on chairs in a single policy (w/ Hongsuk Benjamin Choi Junyi Zhang David McAllister)

Excited to introduce PyRoki ("Python Robot Kinematics"): easier IK, trajectory optimization, motion retargeting... with an open-source toolkit on both CPU and GPU

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: arxiv.org/abs/2505.11709 (1/4)

Was super fun to demo Gemini Robotics @ Google I/O! This was a big effort with the Google DeepMind team including Coline Devin, Sudeep Dasari, and many others. Here's a fun uncut video of me playing with the demo :)

In LLM land, a slow model is annoying. In robotics, a slow model can be disastrous! Visible pauses at best, dangerously jerky motions at worst. But large VLAs are slow by nature. What can we do about this? An in-depth 🧵:

Imitation learning has seen great success, but IL policies still struggle with OOD observations We designed a 3D backbone, Adapt3R, that can combine with your favorite IL algorithm to enable zero-shot generalization to unseen embodiments and camera viewpoints!

Excited to share Flow Matching Policy Gradients: expressive RL policies trained from rewards using flow matching. It’s an easy, drop-in replacement for Gaussian PPO on control tasks.

July has been a big month for Viser! - Released v1.0.0😊 - We did some writing Some demos👇

Inherent biases and imbalances in robot data can make training steerable VLA policies challenging. We introduce CAST, a method to augment datasets with counterfactuals to induce better language following cast-vla.github.io ← paper, code, data, and more available here! 🧵

Robots that navigate grocery stores with semantic 3D maps created by language-augmented gaussian splats

Our latest work performs sim2real dexterous grasping using end-to-end depth RL.

Simulation drives robotics progress, but how do we close the reality gap? Introducing GaussGym: an open-source framework for learning locomotion from pixels with ultra-fast parallelized photorealistic rendering across >4,000 iPhone, GrandTour, ARKit, and Veo scenes! Thread 🧵

How can we standardize conditioning signals in image/video models to achieve the iterative editing & portability that Universal Scene Descriptors provide in graphics? Introducing Neural USD: An object-centric framework for iterative editing & control 🧵