Google DeepMind announces Vision-Language Models as a Source of Rewards paper page: huggingface.co/papers/2312.09… Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting

Monika Wysoczańska Super cool work! We found something similar to the idea of pooling for GEM … github.com/WalBouss/GEM … might be interesting to think about how to merge… overall it looks like an exciting topic! Thanks for sharing 🤩!

super-TL;DR: revival of foveations in vision. They also create a new benchmark for VLMs which is another thing we really need. This looks pretty cool, and after some thinking in 2023, it's also a direction I'm eager to explore in 2024. Can't wait to fully dig into this paper!

[CV] EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models arxiv.org/abs/2401.11739 This paper presents a method for image segmentation using diffusion models. By extracting semantic knowledge from a pre-trained diffusion model, fine-grained segmentation maps

Nice blog post: makeMoE This is a very cool blog post that implements a MoE block + router, discusses initialization and provides very intuitive explanations of how they work huggingface.co/blog/AviSoori1…

pix2gestalt: Amodal Segmentation by Synthesizing Wholes paper page: huggingface.co/papers/2401.14… synthesizes whole objects from only partially visible ones, enabling amodal segmentation, recognition, and 3D reconstruction of occluded objects

VLMs have a resolution problem, which prevents them from finding small details in large images. In this Hugging Face community post, I discuss the ways to solve it and describe the details of MC-LLaVA architecture: huggingface.co/blog/visherati…

Did you know that a pretrained VL foundation model is all you need for open vocabulary localization and segmentation ? ❗️No training needed❗️Just get your favorite model and localize everything! 💎 Pro-tip: try meta-clip All you need to know is below 👇 Grounding x.com/vittoferraricv…

Paper accepted to #CVPR2024! Grounding Everything: Emerging Localization Properties in Vision-Language Transformers Paper: arxiv.org/abs/2312.00878 Demo:huggingface.co/spaces/WalidBo… Code: github.com/WalBouss/GEM With Walid BOUSSELHAM, Felix Petersen, Hilde Kuehne

Are you interested in SOTA compact CLIP models? 🚀🚀 Check out our open-sourced repo for a family of MobileCLIP models, including a ViT-B@224 with 77.2% IN-top1 accuracy. More highlights in 🧵 Paper (appearing in CVPR 2024): arxiv.org/abs/2311.17049 Repo: github.com/apple/ml-mobil…

🚀 Excited to share InstanceDiffusion @CVPR2024! It adds precise instance-level control for image gen: free-form text conditions per instance and diverse location specs—points, scribbles, boxes & instance masks Code: shorturl.at/dtxSW arXiv: shorturl.at/rQS14 1/n

I had so much fun working on LeGrad with Walid BOUSSELHAM, Hendrik Strobelt, and Hilde Kuehne! Demo it here 💻 huggingface.co/spaces/WalidBo…

Join our community-led Geo-Regional Asia Group on Monday, May 27th as they welcome Walid BOUSSELHAM , PhD student at Bonn University, for a presentation on "An Explainability Method for Vision Transformers via Feature Formation Sensitivity." Learn more: cohere.com/events/c4ai-Wa…

Be sure to tune in Monday for our community-led event with speaker Walid BOUSSELHAM , who will be presenting "An Explainability Method for Vision Transformers via Feature Formation Sensitivity." 💻

Come to poster 354 at #CVPR2024's to see our work! 10:30am today, Arch 4A-E "Grounding Everything: Emerging Localization Properties in Vision-Language Transformers" Paper: arxiv.org/abs/2312.00878 Demo:huggingface.co/spaces/WalidBo… Code: github.com/WalBouss/GEM

Today, we’re demonstrating GEM💎: Grounding Everything (in particular grounding VLM transformers) at Demo #13 at #CVPR2024 Walid BOUSSELHAM Hilde Kuehne Vittorio Ferrari github.com/WalBouss/GEM

🚨 New Paper Alert! 🚨 *Mask Inversion* What it does? -> It learns a token representation for a specific region in an image (e.g. a token for a mask, a BBox etc.) project page: walidbousselham.com/MaskInversion/ paper: arxiv.org/abs/2407.20034 code: github.com/WalBouss/MaskI…

🚀 Excited to share our latest research, "MaskInversion: Localized Embeddings via Optimization of Explainability Maps"! Discover how we enhance vision-language models like CLIP for precise image region representation without fine-tuning the underlying foundation model.

🚀We release the dataset, code, models for our "HowToCaption: Prompting LLMs to Transform Video Annotations at Scale," presented at #ECCV2024!🎥📚 🔗Github: github.com/ninatu/howtoca…🔗arXiv: arxiv.org/abs/2310.04900 Anna Kukleva Xudong Hong will go to #NAACL2024 Christian Rupprecht Bernt Schiele Hilde Kuehne