🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Can LLMs keep track of very long conversations? We evaluate 'conversational memory' of LLMs via 3 tasks on our dataset of multi-session multimodal dialogs --> LLMs struggle to remember, reason over history, draw long-range temporal/causal connections arxiv.org/abs/2402.17753 🧵

Can we teach multiple skills to a text-to-image (T2I) model (w/o expensive annotations), while minimizing knowledge conflicts between skills? 🤔 👉SELMA improves T2I models by fine-tuning on automatically generated multi-skill image-text datasets, with skill-specific LoRA expert

Can we adaptively generate training environments with LLMs to help small embodied RL game agents learn useful skills that they are weak at? 🤔 👉 Check out EnvGen, an effective+efficient framework in which an LLM progressively generates and adapts training environments based on

Can we design an efficient & versatile framework to reuse+adapt existing pretrained ControlNets to accurately guide any video/image diffusion model and support diverse controls? 🚨 Introducing Ctrl-Adapter: ➡️ Flexible Compatibility: Adapts any pretrained ControlNet

🎉 Excited that ReGAL has been accepted to ICML Conference #ICML2024! We use LLM-guided refactoring to discover reusable abstractions, which help coding agents avoid repetition+improve reuse! Find me in Vienna (this week & in July)😀 Thanks to my co-authors Archiki Prasad Mohit Bansal

🚨 Check out an exciting set of #ICLR2024 papers/spotlights✨ this week at ICLR 2026 (+workshop on reliable+responsible AI)! Say hi to our awesome students/postdocs (some were/will be on job market) & collaborators, and feel free to ask abt our postdoc openings too** 🙂 🧵👇

Last weekend, I graduated from UNC Computer Science, 10 years after I wrote my first line of code in COMP 116. I'm super grateful to my advisor Mohit Bansal, labmates, intern mentors, and many others. Y'all can see how excited I was as I threw my cap out of the frame to the 2nd floor.

Also, after 10 unforgettable years at Chapel Hill, 2 of those generously sponsored by Apple Scholars in AIML PhD Fellowship, "I'm going to take my talents to Seattle and join Apple AIML". I will continue to do research in efficient and safe AI that generalizes compositionally.

Agentic workflows with LLMs are now getting popular for solving complex tasks! In one of the early works on this topic -- ReConcile, at #ACL2024nlp 🎉 -- we study collaborative model-model interactions w/ confidence-estimation & corrective convincingness btwn diverse LLMs. 🧵👇

🎉Excited to announce that SQ-Transformer is accepted to #ACL2024nlp! We induce systematicity & achieve stronger generalization in Transformers (w/o pretraining on complex data) by structurally quantizing word embedding & regularizing attention outputs. Xiang Zhou Mohit Bansal

🚨New paper👉RACCooN: remove/add/change video content effortlessly/interactively via our MLLM+Video Diffusion (V2P2V) framework with auto-generated descriptions! ▶️ 1. Video-to-Paragraph (V2P): RACCooN first generates well-structured/detailed descriptions of videos with MLLM

🚨 Introducing VideoTree! Captioning + LLMs can perform well on long-video QA, but dense frame captioning leads to inefficiency (redundancy) and sub-optimality (irrelevance). VideoTree addresses these issues & improves LLM-based long-video QA by: ▶️ Structured Video

Check out this work by my labmates on how to make LLMs not overly confident on bad answers. Spoiler 🚨: they made the model less confident on wrong data and more confident on correct ones.

Welcome to UNC NLP! I’m sure you will have a lot of fun doing interesting projects and living in a warmer place full of great college sports matches 😀 Best of luck!

Check out 2 useful updates on CREMA! 🚨 (1a) A new modality-sequential modular training for generalizable and efficient reasoning on video+language+any other modalities by eliminating modality interference. (1b) A novel modality-adaptive early exit strategy allows the model to

Having a great time at #LxMLS in Lisbon + meeting awesome people & exploring the beautiful city 🙂 (highly recommended ML school*) ➡️➡️➡️ Next stop: #ICML2024 in Vienna for MoE tutorial/panel + papers on MAGDi, ReGAL, etc. 👇 (ping me if you want to meet up / chat about

We are going to present MAGDi at #ICML2024. If you are attending, say hi to Elias Stengel-Eskin and Mohit Bansal to know more about this work and PhD/postdoc positions @uncnlp! 🧵👇

🚨 New: my last PhD paper 🚨 Introducing System-1.x, a controllable planning framework with LLMs. It draws inspiration from Dual-Process Theory, which argues for the co-existence of fast/intuitive System-1 and slow/deliberate System-2 planning. System 1.x generates hybrid plans

Check out these other awesome works from my labmates & I will present my poster virtually on Aug22 --> "Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings (arxiv.org/abs/2402.06492)" Detailed thread: x.com/YichenJiang9/s… #ACL2024nlp

🚀 New Paper: RSQ: Learning from Important Tokens Leads to Better Quantized LLMs We show that not all tokens should be treated equally during quantization. By prioritizing important tokens through a three-step process—Rotate, Scale, and Quantize—we achieve better-quantized