🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Happy to share these exciting new results on video synthesis of humans in movement. Arguably, these establish the power of having explicit 3D representations. Popular video generation models like Sora don't do that, making it hard for the resulting video to be 4D consistent.

Boyi Li Wow, this is super cool, Boyi! Such a flashback to Caroline 🐳 and Shiry Ginosar's 'Everybody Dance Now' 😀

Introducing DreamDrive, which combines the complementary strengths of generative AI (video diffusion) and neural reconstruction (Gaussian splatting) to transform any street-view image into a dynamic 4D driving scene! Web: pointscoder.github.io/DreamDrive/ Paper: arxiv.org/abs/2501.00601

I'm happy to post course materials for my class at UC Berkeley "Robots that Learn", taught with the outstanding assistance of Toru. Lecture videos at youtube.com/playlist?list=… Lecture notes & other course materials at robots-that-learn.github.io

Complementing DreamDrive, I am thrilled to introduce STORM, which enables fast scene reconstruction with a single feed-forward model. STORM transforms camera logs into dynamic 3D models - in real time! Web: jiawei-yang.github.io/STORM/ Paper: arxiv.org/abs/2501.00602

Our group at #NVIDIA has a few internship positions available. We welcome talented interns to join our efforts in autonomous driving and VLMs. If you're interested, please email me your CV.

Can Vision-Language Models (VLMs) truly understand the physical world? 🌍🔬 Introducing PhysBench – the first benchmark to evaluate VLMs’ understanding of physics! PhysBench is accepted to #ICLR2025 as an Oral presentation (only 1.8% out of 11k submissions)! 🌐 Project:

Nice to see the progress in interactive task planning. It reminds me of our previous work, ITP, which incorporates both high-level planning and low-level function execution via language. x.com/Boyiliee/statu…

For the first time ever, NVIDIA is hosting an AV Safety Day at GTC - a multi-session workshop on AV safety. We will share our latest work on safe AV platforms, run-time monitoring, safety data flywheels, and more! #AutonomousVehicles #AI at #GTC25 ➡️ nvda.ws/3Xc3xPo

Don’t miss this deep dive into the future of autonomous vehicles! Excited to present about how foundation models are transforming AV technology with Jose M. Alvarez at #GTC25! Check out all the session details below 👇

🚀 New Paper Alert! 🚀 Introducing TULIP 🌷 – a multimodal framework for richer vision-language understanding! A drop-in replacement for CLIP-style models, TULIP learns fine-grained visual details while keeping strong language alignment. 🔗 tulip-berkeley.github.io 🧵👇

At #GTC2025, Jensen unveiled Halos, a comprehensive safety system for AVs and Physical AI. Halos integrates numerous technologies developed by my team NVIDIA, and I was thrilled to help coordinate its launch alongside Riccardo Mariani and many amazing colleagues NVIDIA DRIVE.

Next-gen vision pre-trained models shouldn’t be short-sighted. Humans can easily perceive 10K x 10K resolution. But today’s top vision models—like SigLIP and DINOv2—are still pre-trained at merely hundreds by hundreds of pixels, bottlenecking their real-world usage. Today, we

4K Resolution! Vision is a critical part in building powerful multimodal foundation models. Super excited about this work.

Hallucination is a big challenge in video understanding for any single model. To address this, we introduce Wolf 🐺 (wolfv0.github.io): a mixture-of-experts framework designed for accurate video understanding by distilling knowledge from various Vision-Language Models.

👉🏻 We have released our code and benchmark data at github.com/NVlabs/Wolf. At #GTC 2025, we evaluated the safety and comfort of autonomous driving using Wolf: youtube.com/watch?v=_waPvO….

Nvidia just dropped Describe Anything on Hugging Face Detailed Localized Image and Video Captioning

Introducing the Describe Anything Model (DAM), a powerful Multimodal LLM that generates detailed descriptions for user-specified regions in images or videos using points, boxes, scribbles, or masks. Open-source code, models, demo, data, and benchmark at: describe-anything.github.io

How to equip robot with super human sensory capabilities? Come join us at RSS 2025 workshop, June21, on Multimodal Robotics with Multisensory capabilities to learn more. Featuring speakers: Jitendra MALIK, Katherine J. Kuchenbecker, Kristen Grauman, Yunzhu Li, Boyi Li