Research summary for the last 3 years... 2021: Replace every CNN with a Transformer 2022: Replace every GAN with diffusion models 2023: Replace every NeRF with Gaussian splatting

Check out our latest work. HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models. We propose a diffusion model that can generate realistic 3D human-object interactions (HOIs) driven by textual prompts . Paper: arxiv.org/abs/2312.06553 Project:

Our work OmniControl is accepted in #ICLR2024! Incorporating flexible spatial control signals into a text-conditioned human motion generation model. Project: neu-vi.github.io/omnicontrol/ Code: github.com/neu-vi/omnicon…

Glad to be a recipient of the 2024 Apple Scholars in AI/ML PhD fellowship! Thanks Apple and all my mentors and collaborators! machinelearning.apple.com/updates/apple-…

I will present OmniControl (arxiv.org/abs/2310.08580) at #ICLR2024. ⏰: Tuesday (May 7) 4:30 p.m. (Halle B #54) Come say hi!

Excited to share our recent work HouseCrafter, which can lift a floorplan into a complete large 3D indoor scene (e.g. a house). Our key insight is to adapt a 2D diffusion model to generate consistent multi-view RGB-D images for reconstruction. Paper: arxiv.org/abs/2406.20077

Want to see what your next flat, house or film set could look like in 3D? HouseCrafter can lift a floorplan into a complete 3D indoor scene. neu-vi.github.io/houseCrafter/

We are pleased to announce the availability of Stable Video 4D, our very first video-to-video generation model that allows users to upload a single video and receive dynamic novel-view videos of eight new angles, delivering a new level of versatility and creativity. In

#ECCV2024 We've tamed human motion diffusion models to generate stylized motions. Check out our work SMooDi: Stylized Motion Diffusion Model. One step closer to high-fidelity human motion generation. Paper: arxiv.org/abs/2407.12783 Code: github.com/neu-vi/SMooDi

🔥Today, we announce the MotionLCM-V2, a state-of-the-art text-to-motion model in motion generation quality, motion-text alignment capability, and inference speed. ✍️Blogpost: huggingface.co/blog/wxDai/mot… 💻Code: github.com/Dai-Wenxun/Mot…

Can we robustly track an object’s 6D pose in contact-rich, occluded scenarios? Yes! Our solution, V-HOP, fuses vision and touch through a visuo-haptic transformer for precise, real-time tracking. arXiv: arxiv.org/abs/2502.17434 Project: lhy.xyz/projects/v-hop/

Introducing Stable Virtual Camera: This multi-view diffusion model transforms 2D images into immersive 3D videos with realistic depth and perspective—without complex reconstruction or scene-specific optimization.

🎉Come check out our poster #ICLR2025! SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency 🗓️ Thursday, April 24 ⏰ 3:00 PM – 5:30 PM 📍 Hall 3 + Hall 2B, Poster #112 🧑‍💻 Presented by Chun-Han Yao Huaizu Jiang 🔗 sv4d.github.io

We’ve upgraded Stable Video Diffusion 4D to Stable Video 4D 2.0 (SV4D 2.0), improving the quality of 4D outputs generated from a single object-centric video. While 3D provides a static view of an object’s shape and size; 4D extends this by including time, showing how the object

We revisit the representation in human motion generation, showing that absolute joint coordinates outperform the de facto kinematic-aware, local-relative, and redundant choice. Benefits include: ✅ Easy motion control/editing ✅ Direct generation of SMPL mesh vertices in motion

🌟LMMs e.g. GPT‑o3 can solve spatial tasks from RGBD videos—with strong perception and prompting. 🚀We introduce Struct2D, a method that boosts spatial reasoning in open-source models. Even Qwen-VL-3B + Struct2D outperforms existing 7B models. 📜arXiv: arxiv.org/abs/2506.04220

1) 🚀 From Sketch to Animation! Ever wished your hand-drawn storyboards could come to life? 🎨 Meet Sketch2Anim — our framework that transforms sketches into expressive 3D animations. Presenting at #SIGGRAPH2025 🇨🇦🎉 🔗 Project: zhongleilz.github.io/Sketch2Anim/

🚀 We’ll be hosting a Tutorial on "3D Human Motion Generation and Simulation" at ICCV 2026 in Honolulu, Hawaii! 🌺 📅 Date: October 19, 2026 ⏰ Time: 9:00–16:00 (HST) 🔗 More details & resources: 3dmogen.github.io #AIGC #Simulation #robotics #ComputerVision #ICCV2025