🎉Excited to announce that I will be starting as an Assistant Professor of Computer Science at Rutgers University in Fall 2024! In the meantime, I will spend the next year at Johns Hopkins JHU CLSP as a postdoc with Mark Dredze!

Otter🦦 now knows how to answer a variety of multi-modal real-world questions across a variety of scenarios after we train it how to MIMIC-IT. Can you imagine how an Otter model in your #AppleVisionPro helps your daily life? Check out this! 👇👇👇

Thank you Aran Komatsuzaki for sharing our work! Our project page and code are now available: project page: layoutgpt.github.io code: github.com/weixi-feng/Lay…

What happens if you embody🚶GPT in Street View and provide it with navigation instructions? ✨🤖✨ VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View ✨🤖✨ 💻 map2seq.schumann.pub/vln/velma 📜 arxiv.org/pdf/2307.06082…

Introducing VisIT-Bench, a benchmark for instruction-following vision-language models inspired by real-world use. Aiming for better multimodal chatbot evaluation with an automated ranking system🚀 📜 arxiv.org/abs/2308.06595 🌐 visit-bench.github.io ✍️laion.ai/blog/visit_ben… 🧵

Our mmc4 is accepted to #NeurIPS2023 D&B Track. Thanks again to my amazing collaborators, especially Jack Hessel ! See you guys in New Orleans 😆

I am on the academic job market for 2024 Fall, with a research focus on Improving the Factuality and Reasoning ability of Large Language Models. Open to opportunities worldwide. Feel free to reach out if your department has open positions. liangmingpan.com Thanks!

Happy to share VisIT-Bench's acceptance to #NeurIPS2023 D&B! As multimodal chatbots rise, real-world instruction following evaluation is crucial. VisIT-Bench's auto-eval aligns closely with human preferences. We've updated the arXiv & leaderboard; researchers, add your models!📢

We're proud to announce the winners of our Rising Stars in Machine Learning program, designed to support researchers from underrepresented groups. Pictured clockwise: Megha Srivastava, Han Shao, Wanrong Zhu, Laixi Shi, Zhijing Jin, Yutong Bai, Weijia Shi, and Sanae Lotfi.

GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation paper page: huggingface.co/papers/2311.07… present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone

How does LLM walk in the New York City? Here’s an interactive demo by Raphael Raphael Schumann : map2seq.schumann.pub/velma/demo/ Talk to them at #NeurIPS2023 Congrats Raphael Schumann Wanrong Zhu @NeurIPS Weixi Feng on the accepted #AAAI2024 paper on LLMs for decision making. arxiv.org/abs/2307.06082

Finally flying to NOLA for #NeurIPS2023 after some unexpected delay. It’s my first in-person conference and I am excited to present LayoutGPT with my coauthors on Wed. 5-7pm, poster session 4. Looking forward to meet new friends and chat about image gen., LLMs, or anything else!!

Our NeurIPS paper about understanding LLMs' in-context learning ability will be presented by my amazing coauthor Wanrong Zhu @NeurIPS at poster session 2 on Tuesday, 5:15-7:15 pm. Please drop by if you are interested! I won't be there in person but I'm happy to talk more about it online😀

Just arrived at #NewOrleans for #NeurIPS2023 (a looong flight from emnlp)! Tomorrow I'll present Multimodal C4 at poster session 1 (#310, Tue 10:45am). Welcome everyone 😄

Congratulations Drs Tsu-Jui Fu Alon Albalak Wanrong Zhu @NeurIPS Yi-Lin (Pascal) Tuan on your graduation! The world of artificial intelligence is better because of your contributions. To your continued success! 🚀

🚨 Introducing “MMSci: A Multimodal Multi-Discipline Dataset for PhD-Level Scientific Comprehension” arxiv.org/abs/2407.04903 🧐Have current multimodal LLMs achieved PhD-level intelligence across diverse scientific disciplines? Are they ready to become AI scientific assistants?

I am attending #COLM2024 in Philly! Will present our paper “List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs” on Monday morning ⏰ Come and chat if you are interested in multimodal LLMs, synthetic data and training recipes!

📢 Adobe Research Document Intelligence Lab is hiring PhD research interns! 🚀 If you're passionate about vision-language challenges, MLLM, and cutting-edge multimodal research, we'd love to hear from you! Drop me an email and apply here: careers.adobe.com/us/en/job/R150…