Boston Dynamics collaborated with NVIDIA to demonstrate DextrAH-RGB, a workflow for dexterous grasping from stereo RGB input. The end-to-end policy for Atlas robot, trained entirely in NVIDIA Isaac Lab, transfers zero-shot from simulation to the real robot.

Teleoperation is slow, expensive, and difficult to scale. So how can we train our robots instead? Introducing X-Sim: a real-to-sim-to-real framework that trains image-based policies 1) learned entirely in simulation 2) using rewards from human videos. portal-cornell.github.io/X-Sim

We find keypoint trajectories to be a powerful interface between VLM planning & RL control VLM: Generates object + hand motion plan from a task prompt & RGB-D image (perception + commonsense) RL policy: Conditioned on the plan, learns low-level dexterous control (0-shot sim2real)

The Dex team at NVIDIA is defining the bleeding edge of sim2real dexterity. Take a look below 🧵 There's a lot happening at NVIDIA in robotics, and we’re looking for good people! Reach out if you're interested. We have some big things brewing (and scaling :)