🚀 (1/6) Excited to release LLaVA-OneVision (OV)! An open, large multimodal model that excels across single-image, multi-image, and video tasks, while effortlessly unlocking new emerging capabilities with task transfer Paper: arxiv.org/abs/2408.03326 Blogs: llava-vl.github.io/blog/

I wrote a blogpost "On the speed of ViTs and CNNs". Addresses the following concerns I often hear: - worry about ViTs speed at high resolution. - how high resolution do I need? - is it super important to keep the aspect ratio? I think Yann LeCun might like it too! Link below

New Video: How to write an okay research paper. Reviewers all agree! Sasha Rush's papers are "reasonably structured" and "somewhat clear, despite other flaws". youtu.be/qNlwVGxkG7Q?si…

I made a github repository exploring this question. github.com/mickaelChen/To… We visualize the embeddings inside a DiT and observe how it works around this issue. We also build a very simple toy experiments to see the limits of the method. tldr: not a big issue, but kind of ugly.

I’ll be giving a talk on « vision in the age of LLMs » on Monday 11:00 cet which is open to public via the link below:

I'm thrilled that the paper Kaiqing Zhang and I co-authored has made it onto the reading list for this course! ambujtewari.com/LLM-fall2024/ Any questions would be appreciated.

LLaVA-OneVision-Chat🤖💬 significantly improves the chat experience of LLaVA-OneVision, as demonstrated on 5 multimodal task! 🌟, achieved by preference alignment & self-critic feedback 📊 Doc: github.com/LLaVA-VL/LLaVA…

Are you writing an ICLR paper, and wondering... * What level of detail should the background have? * Why do people read related work? * How do I convey convincing results? I have some mostly satisfactory answers to these questions. youtu.be/qNlwVGxkG7Q?si…

🔥 Paper Drop 🔥 What can we understand by peering inside vision-language models (VLMs) like LLaVA? We show that image representations inside VLMs can be directly interpreted and edited in the language space, and we apply our findings to mitigate hallucinations!

It looks like there are some efforts to improve on vanilla attention in transformers ... arxiv.org/abs/2410.02703 Selective Attention arxiv.org/abs/2410.11842… MHA as Mixture of Heads Attention arxiv.org/abs/2410.05258 Differential Transformer

Tips for Writing a Research Paper using LaTeX github.com/guanyingc/late…

I'm happy to post course materials for my class at UC Berkeley "Robots that Learn", taught with the outstanding assistance of Toru. Lecture videos at youtube.com/playlist?list=… Lecture notes & other course materials at robots-that-learn.github.io

I am teaching a class on #FoundationalModels for #robotics and Scaling #DeepRL algorithms. This class expands on last year's class and my generalist robotics policies tutorial and code. I plan to share the lectures and code assignments. Starting with the first lectures below.

Got talked into giving a DeepSeek talk this afternoon simons.berkeley.edu/workshops/llms… Not sure I have anything new to say here! But good excuse for me to read all the blogs.

Re-sharing my 3.5 hr lecture on large language models as some might be interested. Will post updated lectures/materials/news in coming weeks. Stay tuned! Link: youtu.be/2yjzZfDQxy8 Topics: 0:00 Basics of language models 2:30 Word2vec 16:27 Transfer Learning 19:23 BERT

New blog post: let's talk about latents! sander.ai/2025/04/15/lat…

What do you think are the most important improvements of the 2017 transformer architecture?

As expected, that was popular. Here is my attempt at consolidating all the answers into a list. - Prenorm: normalization in the residual blocks before the attention operation and the FFN respectively - GQA (Group Query Attention): more Q than (K, V)

🚀 Details of the #CVPR2025 award candidate papers are out. 14 of 2967 accepted papers made the list, spanning 3D vision, embodied AI, VLMs/MLLMs, learning systems, and scene understanding. 3D vision leads with the most entries. I collected the TL;DR, paper, and project links👇

Giving your models more time to think before prediction, like via smart decoding, chain-of-thoughts reasoning, latent thoughts, etc, turns out to be quite effective for unblocking the next level of intelligence. New post is here :) “Why we think”: lilianweng.github.io/posts/2025-05-…