🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

For The Verge and New York Magazine, I spoke with the people teaching the chatbots how to chat theverge.com/features/23764…

With Nadine Chang we introduce a new task, Labeling Instruction Generation, to address missing publicly available labeling instructions from large-scale, visual datasets. Timely for new article from Josh Dzieza in The Verge and New York Magazine arXiv preprint: arxiv.org/abs/2306.14035

Worked hard, learned a lot, and met a great team during the first-ever competition for generative everyday sounds at DCASE Challenge, Task 7 Foley sound! Keunwoo Choi Jaekwon Im KeisukeImoto Shinnosuke Takamichi / 高道 慎之介 Yuki Okamoto Mathieu Lagrange, Brian McFee.

Our lab at Carnegie Mellon University is recruiting 18+ #misophonics in the Pittsburgh area for an in-person study on #misophonia! 🔊 1.5-2hrs at $12/hr 🔊 Listen to everyday sounds while being monitored by our team Please share with your networks! More info below

1/ “Prepare my evening tea” How can we enable embodied agents to execute open-domain instructions and personalized requests? Excited to share our work HELPER, an open-ended, instructable agent with memory-augmented LLMs, accepted at #EMNLP2023! 🤖 helper-agent-llm.github.io

I’ll be at #NeurIPS all week (11-17) and will be presenting Brain Dissection⬇️ Poster #418 Thursday! Also am very excited to chat about: - LLM/VLM/MLLM for decision making and embodied agents - personalized and memory-augmented agents - neuronAI Reach out if you want to chat!

How can we get VLMs to move their eyes—and reason step-by-step in visually grounded ways? 👀 We introduce ViGoRL, a RL method that anchors reasoning to image regions. 🎯 It outperforms vanilla GRPO and SFT across grounding, spatial tasks, and visual search (86.4% on V*). 👇🧵

We ran a behavioral analysis of vanilla RL on small base VLMs and found: – Minimal reference to fine-grained visual cues – Poor exploration of visual regions – Over-reliance on abstract, ungrounded heuristics—“look-once & guess” behavior ❌

Why does vanilla RL fail here? RL can only amplify base behaviors: - Pretrained VLMs bias toward abstract scene references not region analysis - Accuracy-only rewards reinforce this We argue: grounding each thought shifts models toward iterative, perceptually guided reasoning.

Our grounded RL recipe: – Warm-start on MCTS-generated (x, y)-anchored chains – GRPO RL to reinforce correct + grounded steps – Multi-turn RL for zooming into visual detail Result? A model that iteratively attends, verifies, and self-corrects, grounding thoughts in the scene.

ViGoRL delivers gains across tasks: – V*Bench (Search): 86.4%, outperforms GPT-4o & tool-use – SAT-2: +12.9 pts, BLINK: +2.0 pts vs GRPO – ScreenSpot-Pro: +3.3 pts, VisualWebArena: +3.0 pts (vision-only) – Removing grounding hurts VLMs that ground their reasoning see better.

Why does grounding help? Explicit grounding boosts helpful visual behaviors lacking in vanilla RL: - More visual region exploration (3.5 vs 1.8) - Frequent visual verification (0.39 vs 0.14) - Does backtracking (0.47) RL + Grounding = richer reasoning 🧠

We show spatial grounding in reasoning is accurate and aids human interpretability! Human evaluations confirm: - High accuracy in visual grounding of thoughts - Grounded reasoning steps significantly help human understanding of reasoning (especially when grounding is correct!)🧐

Cognitive scientists have long studied deictic reference and feature binding for how humans use eye movements as spatial pointers to bind abstract variables to perceptual content, enabling compositional and goal-directed reasoning. ViGoRL provides evidence for this in VLMs!

👉 Paper: arxiv.org/abs/2505.23678 👉 Project page: visually-grounded-rl.github.io This was an amazing collaboration with co-authors Snigdha Saha, Naitik Khandelwal, and Ayush Jain, together with faculty Katerina Fragkiadaki, Aviral Kumar, and Academic Mercenary.

1/6 🚀 Excited to share that BrainNRDS has been accepted as an oral at #CVPR2025! We decode motion from fMRI activity and use it to generate realistic reconstructions of videos people watched, outperforming strong existing baselines like MindVideo and Stable Video Diffusion.🧠🎥

2/6 Instead of directly generating the video from fMRI data, we first decode the motion, then use the motion to generate the video.

3/6 Our method leads to more accurate object-level motion decoding. We achieve significantly lower end-point error than baselines that try to generate videos directly (e.g., MindVideo, Stable Video Diffusion).

4/6 BrainNRDS also achieves better video reconstruction with an fMRI-generated initial frame.

5/6 We also show that dynamic information isn’t just useful for generation. It is key to understanding brain activity as well. We observe that video models predict brain responses to dynamic scenes better than image models, especially in visual and somatosensory cortices.