Excited to share LRM (large reconstruction model), which views 2D->3D as a multimodal problem and learns transformer from large-scale data. A cornerstone of our 3D foundation model efforts at Adobe Research. Nice work by intern (also upcoming full-time) Yicong Hong.

🔥Check out 🪄Lumos, our open general language agent! Lumos has features: 🧩General modular framework 🌍Tuned with diverse agent training data 🚀Strong perf vs GPT/larger open agents MOSAIC uclanlp Ai2 📝: arxiv.org/abs/2311.05657 💻: github.com/allenai/lumos (1/N)

📢 📽✍️We introduce VideoCon, a video-text dataset for training SOTA alignment model. It resolves a typical issue in video-text alignment models that struggles with robustness. w/ leah bitton, Idan Szpektor, Kai-Wei Chang , Aditya Grover video-con.github.io 🧵 1/

How to best leverage your pre-trained language model for keyphrase generation?📇Still directly fine-tuning BART/T5 and using greedy decoding?⚠️Check out our #EMNLP2023 paper for why you may or may not want to do that (1/N)

VL models fail at spatial reasoning, but biases in benchmarks (dogs are usually UNDER tables) mask even worse capabilities. 📢 A new benchmark at #EMNLP2023 without this bias (yes, we put a dog on a table) in What’s up with VL models? arxiv.org/pdf/2310.19785… Jack Hessel uclanlp

Arrived at #NeurIPS2023! Looking forward to meeting old and new friends! We will present our work on teaching models to ground by descriptions (DesCo) on Wed morning. Also checkout our demo huggingface.co/spaces/zdou083…!

#NeurIPS2023 Stop by our poster at Great Hall & Hall B1+B2 (level 1) #1925 (today 10:45-12:45) to chat about object recognition with language descriptions! Paper: openreview.net/pdf?id=WKJDGfU… Code: github.com/liunian-harold… Demo: huggingface.co/spaces/zdou083…

I am honored to be nominated by SIGDAT (the org that oversees EMNLP) to run for VP-elect with other awesome candidates who share the goal of improving our community. Please check your email to vote by 3/24.🗳️ See details: bit.ly/3ItRc0S

Congrats 🎉 to the newly titled Dr. Lu Pan Lu on defending his thesis about mathematical reasoning with language models"! 🧮 Pan has published a series of works on quantifying and improving math and scientific reasoning ability in LLMs. Some highlights:

Checkout this awesome work on early pandemic detection by Tanmay Parekh and his team using NLP tools!

🔍🚨How to improve multilingual performance on structured prediction tasks? Excited to share our latest work CLaP - a label projection technique utilizing LLMs to do contextualized machine translation, improving two tasks in 47 languages including 10 extremely low-resource ones!

✨New Paper Alert✨ Excited to introduce ExPO, an extremely simple method to boost LLMs' alignment with human preference, via weak-to-strong model extrapolation 👇 #LLMs #MachineLearning #NLProc #ArtificialIntelligence #AI

I am thrilled to share that I will join the Department of Computer Science and Engineering at Texas A&M University as an Assistant Professor in Fall 2024. Many thanks to my advisors, colleagues, and friends for their support and help. I'm really excited about the new journey at College Station!

How to pick a good number of visual tokens? Too few, you have poor performance; too many, you need quadratically more compute. In this work, we introduce a model that works with an elastic number of tokens. arXiv: arxiv.org/abs/2405.19315

Very excited that we have a similar work almost simultaneously. Some comparisons are: our model enable any number of visual tokens under a predefined maximum accommodating various computational constraints. ... 1/3

Proposing Ctrl-G, a neurosymbolic framework that enables arbitrary LLMs to follow logical constraints (length control, infilling …) with 100% guarantees. Ctrl-G beats GPT4 on the task of text editing by >30% higher satisfaction rate in human eval. arxiv.org/abs/2406.13892

Introducing ♾OmniJARVIS, our latest venture to #AgentGPT, or vision-language-action (VLA) models for open-world instruction-following agents 🦾🕹️ tuning in 👉omnijarvis.github.io by Team CraftJarvis, 🤿⏬

📢 I’m recruiting PhD students UVA Computer Science for Fall 2025! 🎯 Neurosymbolic AI, probabilistic ML, trustworthiness, AI for science. See my website for more details: zzeng.me 📬 If you're interested, apply and mention my name in your application: engineering.virginia.edu/department/com…

Today's the day! sora.com Included with ChatGPT Plus/Pro!

Sora is here for Plus and Pro users at no additional cost! Pushing the boundaries of visual generation will require breakthroughs both in ML and HCI. Really proud to have worked on this brand new product with Bill Peebles Rohan Sahai Connor Holmes and the rest of the Sora team!