✨ Introducing 𝐎𝐩𝐞𝐧𝐕𝐋𝐀 — an open-source vision-language-action model for robotics! 👐 - SOTA generalist policy - 7B params - outperforms Octo, RT-2-X on zero-shot evals 🦾 - trained on 970k episodes from OpenX dataset 🤖 - fully open: model/code/data all online 🤗 🧵👇

Cambrian-1 🪼 Through a vision-centric lens, we study every aspect of building Multimodal LLMs except the LLMs themselves. As a byproduct, we achieve superior performance at the 8B, 13B, 34B scales. 📄arxiv.org/abs/2406.16860 🌎cambrian-mllm.github.io 🤗huggingface.co/nyu-visionx

Introduce Open-𝐓𝐞𝐥𝐞𝐕𝐢𝐬𝐢𝐨𝐧🤖: ⁣ We need an intuitive and remote teleoperation interface to collect more robot data. 𝐓𝐞𝐥𝐞𝐕𝐢𝐬𝐢𝐨𝐧 lets you immersively operate a robot even if you are 3000 miles away, like in the movie 𝘈𝘷𝘢𝘵𝘢𝘳. Open-sourced!

PhysAvatar has been accepted into #ECCV2024! 🎉 We create virtual avatars with realistic garment dynamics and lighting, with a physics simulator in the loop.

Introduce Mobility VLA - Google's foundation model for navigation - started as my intern project: - Gemini 1.5 Pro for high-level image & text understanding - topological graphs for low-level navigation - supports multimodal instructions co-lead Zhuo Xu, Lewis Chiang, Jie Tan

At #RSS2024, we are excited to share AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries, which will be presented at RSS 2024. AutoMate is the first simulation-based framework for learning specialist (part-specific) and generalist (unified) assembly

Want to #WalkTheDog in the metaverse? In our project at #SIGRAPH2024 with Sebastian Starke , Yuting Ye and Olga Sorkine-Hornung, we develop an approach to learn a common 1D phase manifold from motion datasets across different morphologies, *without* any supervision (1/2)

We'll present "Neural Control Variates with Automatic Integration" at the Monte Carlo for PDE session in Mile High 4 at 11:25 tomorrow. Come to our talk to learn more about creating control variates function with arbitrary neural networks! #SIGGRAPH2024

My first SIGGRAPH at #SIGGRAPH2024 ! Chen-Hsuan Lin Jiashu Xu Donglai Xiang will show 3D scene generation in real time from scratch along with other 10 RTL participants. Join us at 6p today!

very cool🔥

Congrats Tsung-Yi and well deserved 🎉

kudos to the team!

Introducing OFT—an Optimized Fine-Tuning recipe for VLAs! Fine-tuning OpenVLA w/ OFT, we see: -25-50x faster inference ⚡️ -SOTA 97.1% avg SR in LIBERO 💪 -high-freq control w/ 7B model on real bimanual robot -outperforms π₀, RDT-1B, DiT Policy, MDT, Diffusion Policy, ACT 🧵👇

Excited to share our work: Gaussian Mixture Flow Matching Models (GMFlow) github.com/lakonik/gmflow GMFlow generalizes diffusion models by predicting Gaussian mixture denoising distributions, enabling precise few-step sampling and high-quality generation.

Curious about how cities have changed in the past decade? We use MLLMs to analyse 40 million Street View images to answer this. Do you know that "juice shops became a thing in NYC" and "miles of overpasses were painted BLUE in SF"? More at→boyangdeng.com/visual-chronic… (vid ↓ w/ 🔊)