We introduce the first multimodal instruction tuning dataset: 🌟MultiInstruct🌟 in our 🚀#ACL2023NLP🚀 paper. MultiInstruct consists of 62 diverse multimodal tasks and each task is equipped with 5 expert-written instructions. 🚩arxiv.org/abs/2212.10773🧵[1/3]

Struggling with catastrophic forgetting when updating your model? 🤯 Check out our latest work to appear at Findings of 🌟#ACL2023NLP🌟! We extensively study the *classifier drift* issue in continual learning and introduce an effective framework for this problem. 📌paper at

Congrats to The Sanghani Center at Virginia Tech core faculty Lifu Huang, alum Aditya, Ph.D. student Barry Menglong Yao and their collaborators who garnered this award SIGIR 2025!

Very excited and honored to receive the *Best Paper Award Honorable Mention* from SIGIR'2023! Congrats to Barry Menglong Yao and all collaborators and welcome to check out the first end-to-end multimodal fact-checking benchmark (dl.acm.org/doi/pdf/10.114…)

👋for Team HokieBot, awarded 3rd Place in the science innovation category Amazon Science #AlexaPrize SocialBot Grand Challenge 5! 🎉 to The Sanghani Center at Virginia Tech Virginia Tech Computer Science Ph.D. students Ying Shen (team leader), Minqian Liu, zhiyang xu, Barry Menglong Yao, and faculty advisor Lifu Huang.

We are interviewed by Research Voyage about our Best Paper Award Honorable Mention paper at SIGIR 2023. Here is the interview blog post: researchvoyage.com/barry-yao-best…. Our paper: dl.acm.org/doi/10.1145/35…

Today we officially release ✨Vision-Flan✨, the largest human-annotated visual-instruction tuning dataset with 💥200+💥 diverse tasks. 🚩Our dataset is available on Huggingface huggingface.co/datasets/Visio… 🚀 For more details, please refer to our blog vision-flan.github.io/index.html

Our new work ✨The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models✨ is accepted to #EMNLP2023. Inspired by the human cognitive process, we propose SOCRATIC QUESTIONING, a divide-and-conquer style algorithm that mimics the 🤔recursive thinking process.

Our entity linking work has been accepted by #EACL2024. Check our work: Ameli: Enhancing Multimodal Entity Linking with Fine-Grained Attributes (arxiv.org/pdf/2305.14725…). Congratulations to all collaborators! The dataset, code, and checkpoints will be released soon.

Vision-Flan Scaling Human-Labeled Tasks in Visual Instruction Tuning Despite vision-language models' (VLMs) remarkable capabilities as versatile visual assistants, two substantial challenges persist within the existing VLM frameworks: (1) lacking task diversity in pretraining

🚀 Excited to introduce my internship work at Apple MLR : Many-to-many Image Generation with Auto-regressive Diffusion Models (arxiv.org/abs/2404.03109). Exploring the paradigm for domain-general multi-image to multi-image generation.

I will start my Applied Scientist internship at Amazon AGI team this summer!

🚨 New paper alert! We introduce InterleavedBench📚, the first comprehensive evaluation benchmark for interleaved text-and-image generation, as well as InterleavedEval🔍, a powerful GPT-based evaluator that supports multi-aspect assessment. arXiv: arxiv.org/abs/2406.14643 (1/n)

Excited to share that our paper "Error-driven Data-efficient Large Multimodal Model Tuning" has been accepted to #ACL2025 !🎉 We propose an error-driven tuning framework for efficiently adapting large multimodal models (LMMs) to newly emerging tasks without requiring extensive

I will be in Seattle this summer working as an Applied Scientist intern at Amazon Alexa AI. If you are around, let us catch up!