How about foundation models having a nice conversation ?

An excellent summary of some of the achievements of our Team in 2022: ai.googleblog.com/2023/02/google…. Performers were recently applied to make on-robot inference of Transformer-based models feasible. Is <10ms on-robot latency for 8.3M-parameter Transformer model possible ? Yes, it is.

Presenting Performer-MPC, an end-to-end learnable robotic system that combines several mechanisms to enable real-world, robust, and adaptive robot navigation with real-time, on-robot transformers. Read more about this system → goo.gle/3IRNNc6

PaLM-E or GPT-4 can speak in many languages and understand images. What if they could speak robot actions? Introducing RT-2: robotics-transformer2.github.io our new model that uses a VLM (up to 55B params) backbone and fine-tunes it to directly output robot actions!

Excited to present RT-2, a large unified Vision-Language-Action model! By converting robot actions to strings, we can directly train large visual-language models to output actions while retaining their web-scale knowledge and generalization capabilities! robotics-transformer2.github.io

Multi-modal Transformers in action on robots give robots unprecedented capabilities to reason about the world. Check out RT-2, our new robotic model that makes it happen !

The concept of the associative memory is an elegant idea, strongly motivated by how we humans memorize and with a beautiful mathematics behind it. Now there is a NeurIPS workshop on it :) amhn.vizhub.ai

To produce truly capable robots, two fundamental challenges must be addressed: 🔘 Improving their ability to generalize their behavior to novel situations 🔘 Boosting their decision-making speed We deliver critical improvements in both areas. ↓ dpmd.ai/advanced-robot…

1️⃣ Our new system SARA-RT converts Robotics Transformer models into more efficient versions using a novel method: “up-training.” This can reduce the computational requirements needed for on-robot deployment, increasing speed while preserving quality. dpmd.ai/advanced-robot…

🔵 When we applied SARA-RT to our state-of-the-art RT-2 model, the best versions were 10.6% more accurate & 14% faster after being given a short history of images. We believe this is the first scalable attention mechanism to offer computational improvements with no quality loss.

To reach a future where robots could help people with tasks like cleaning and cooking, they must learn to make accurate choices in a fast-changing world. By building on our RT-1 and RT-2 models, these projects take us closer to more capable machines. ↓ dpmd.ai/advanced-robot…

We are shaping the future of advanced Robotics. Introducing AutoRT, SARA-RT and RT-Trajectories to address the notoriously difficult problems of generalization and speed of Transformer-based models in Robotics. x.com/GoogleDeepMind…

New Google DeepMind blog post covering 3 recent works on AI + robotics! 1. Auto-RT scales data collection with LLMs and VLMS. 2. SARA-RT significantly speeds up inference for Robot Transformers. 3. RT-Trajectory introduces motion-centric goals for robot generalization.

SARA-RT (fast-inference Robotics Transformers) presented at the Best Papers in Robotics Manipulation Session at #ICRA2024 in Yokohama, with live robotic demo from New York. Great job Team !

SARA-RT winning The Best Robotic Manipulation Paper Award at #ICRA2024 in Yokohama, the biggest Robotics conference in history (6K+ attendees). The only paper at the conference with final presentation complemented with live robotic demo. Congratulations Team !

"Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity" winning The Best RoboCup Paper Award at #IROS2024 in Abu Dhabi, UAE. Congratulations to the Team !

This is so far a pretty unusual year. Two best award papers in two top Robotics conferences: ICRA 2024 and IROS 2024, within last 5 months. I am blessed to work with amazing people...

Very proud of our Google DeepMind Robotics Team ! Big milestone and an accomplishment of the entire Team.