👁️ 𝐂𝐚𝐧 𝐕𝐢𝐬𝐢𝐨𝐧 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 (𝐕𝐋𝐌𝐬) 𝐈𝐧𝐟𝐞𝐫 𝐇𝐮𝐦𝐚𝐧 𝐆𝐚𝐳𝐞 𝐃𝐢𝐫𝐞𝐜𝐭𝐢𝐨𝐧? Knowing where someone looks is key to a Theory of Mind. We test 111 VLMs and 65 humans to compare their inferences. Project page: grow-ai-like-a-child.github.io/gaze/ 🧵1/11

Gaze has been on my mind for a long time. In real life, we don’t use mouse cursors—we use gaze, head turns, and gestures to refer nonverbally. We (GrowAIlikeAChild) ask: can VLMs interpret gaze like humans do? Spoiler: they mostly chase head direction, not eye gaze. Having

What is the best testbed for human-like cognition in machines?🤔 Hardly anything comes better than gaze understanding. Humans are extremely good at gaze reading 👁️ , and this ability develops extremely early in childhood. Here, we tested 111 VLMs and 65 humans through

New Paper Alert ‼️Current VLMs completely fail human gaze understanding 🙀 and scaling does NO help ‼️ However, humans, since an extremely age 🧒, are extremely sensitive to other people's gaze 🙄 👀 No mentors, no labs, only pre-doc students, 111 VLMs, and we did it 😎

Excited to share project specifying a research direction I think will be particularly fruitful for theory-driven cognitive science that aims to explain natural behavior! We're calling this direction "Naturalistic Computational Cognitive Science"

#ICML #cognition #GrowAI We spent 2 years carefully curated every single experiment (i.e. object permanence, A-not-B task, visual cliff task) in this dataset (total: 1503 classic experiments spanning 12 core cognitive concepts). We spent another year to get 230 MLLMs evaluated

💥We are excited to share that our #Video2Animation feature is now live at myolab.ai's discord servers. We're giving away massive 𝐟𝐫𝐞𝐞 credits for early users. Head now to 👉 discord.gg/Gc9hapPgQg

The final version of this paper has now been published in open access in the Journal of Memory and Language (link below). This was a long-running but very rewarding project. Here are a few thoughts on our methodology and main findings. 1/9