Learning style from a single image is difficult, but what if you had access to an **image pair** instead? I’m excited to share our #SIGGRAPHASIA2024 work PairCustomization, on customizing text-to-image models with a single image pair!! project page: paircustomization.github.io

David and Aleks are going to present our work on making sense of SDS this Friday at #NeurIPS2024! Please check it out if you are around!

Check out our new project on making 3D illusions! Here is my favorite example, "Finding Nemo"

As my kids are singing APT non-stop these days, I did a bit of reverse engineering of the APT music video and tried to understand why the MV is so addictive. Here is what I learned.

- Humans and Structure from Motion - We jointly reconstruct 3D humans, scene point cloud, and cameras from images captured with sparse uncalibrated cameras. ✨Enjoy reading & happy holidays✨ Project page: muelea.github.io/hsfm

github.com/NVIDIA/Cosmos Cosmos is a developer-first platform designed to help physical AI builders accelerate their development. It has pre-trained world foundation models (diffusion & autoregressive) in different sizes and video tokenizers. They are open models with permissive

Introducing "AutoPresent: Designing Structured Visuals From Scratch". We employ code generation to create structured, high-quality presentation slides from scratch! 📄 arxiv.org/abs/2501.00912 🤗 huggingface.co/spaces/JiaxinG… 🔗 github.com/para-lost/Auto… Berkeley AI Research Language Technologies Institute | @CarnegieMellon

Decentralized Diffusion Models UC Berkeley and Luma AI introduce Decentralized Diffusion Models, a way to train diffusion models on decentralized compute with no communication between nodes. Leveraging the associative property of the marginal flow (due to it being a linear

Decentralized Diffusion Models power stronger models trained on more accessible infrastructure. DDMs mitigate the networking bottleneck that locks training into expensive and power-hungry centralized clusters. They scale gracefully to billions of parameters and generate

Very excited to share Stable Virtual Camera, a generalist diffusion model for view synthesis: stable-virtual-camera.github.io It scales well with data, and works out-the-box for different NVS tasks. Code and 🤗 demo are released! 🧵(1/N)

Introducing St4RTrack!🖖 Simultaneous 4D Reconstruction and Tracking in the world coordinate feed-forwardly, just by changing the meaning of two pointmaps! st4rtrack.github.io

Check out this super cool project that enables humanoid learning from casual human video!! Very exciting moment!

Excited to introduce PyRoki ("Python Robot Kinematics"): easier IK, trajectory optimization, motion retargeting... with an open-source toolkit on both CPU and GPU

Are embeddings from decoder-only transformers, like Mistral and Qwen, worse than T5 embedding for text-to-image models? Not really! We find that when the embeddings from proper layers are extracted and normalized, they can outperform T5 embedding.

What would a World Model look like if we start from a real embodied agent acting in the real world? It has to have: 1) A real, physically grounded and complex action space—not just abstract control signals. 2) Diverse, real-life scenarios and activities. Or in short: It has to

Some problems can’t be rushed—they can only be done step by step, no matter how many people or processors you throw at them. We’ve scaled AI by making everything bigger and more parallel: Our models are parallel. Our scaling is parallel. Our GPUs are parallel. But what if the

Training diffusion-style policies with RL can be as easy as training Gaussian policies. We introduce Flow Policy Optimization (FPO) — bringing flow matching into the policy gradient world. Multimodal. Sampling-agnostic. More expressive than Gaussians.