Handling long context in LLMs is expensive, but can we cut the cost by learning them offline for a specific set/genre of documents?

Introducing LLoCO, our new technique that learns documents offline through context compression and in-domain finetuning using LoRA, which archives…

Thanks Aran for sharing!
AI feedbacks will enable autonomous evaluation and improvement of language agents at scale. We have a thread here if you wanna learn more :)

twitter.com/pan_jiayipan/s…

Autonomous Evaluation and Refinement of Digital Agents

Improves WebArena's GPT4 SotA agent by 30%+ and CogAgent in iOS by 75% without any extra supervision but only a VLM-based evaluator

repo: github.com/Berkeley-NLP/A…
abs: arxiv.org/abs/2404.06474

GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications

Presents a runtime for LLMs with an intuitive undo and damage confinement abstractions, enabling the safer deployment of LLM agents in practice

repo: github.com/ShishirPatil/g…
abs:…

🔥 Not All Tokens Are What You Need!
🚀 Releasing the Rho-1 series, including the first 1B LLM to hit 40.6% on MATH.

Rho-1 introduces Selective Language Modeling ( #SLM ) for token-level pretraining data selection.

Thanks to AK and Aran Komatsuzaki for sharing our work!

We have released the training corpus, models and a lot of multilingual evaluation benchmarks in Hugging Face: huggingface.co/collections/Hi…

We hope this project begins a wave of multilingual models for the medical domain!

Thanks Aran for sharing our work!

This is a survey paper I’ve been thinking about for a long time, as we have seen an increasing need for synthetic data. As we will probably run out of fresh tokens soon, the audience of this paper should be everyone who cares about AI progress.

AssemblyAI presents Conformer-1: Robust ASR via Large-Scale Semisupervised Bootstrapping

Presents an end-to-end ASR model trained on 570k hours of speech data

arxiv.org/abs/2404.07341

Google presents Best Practices and Lessons Learned on Synthetic Data for Language Models

Provides an overview of synthetic data research, discussing its applications, challenges, and future directions

arxiv.org/abs/2404.07503

From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

Several LLMs (e.g., GPT-4) perform on par w/ supervised methods like Random Forest on regression

repo:github.com/robertvacarean…
abs: arxiv.org/abs/2404.07544

Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

arxiv.org/abs/2404.07613

Thanks Aran for sharing!! 🤗🤗

Let’s work together towards general purpose computer agent this time again, with multimodal language model agent.

(Will have an official post later.

ByteDance presents InfiCoder-Eval

InfiCoder-Eval comprises 270 carefully picked high-quality StackOverflow questions, covering 18 programming languages, for evaluating code LLMs

proj: infi-coder.github.io/inficoder-eval/
abs: arxiv.org/abs/2404.07940

Microsoft presents Rho-1: Not All Tokens Are What You Need

RHO-1-1B and 7B achieves SotA results of 40.6% and 51.8% on MATH dataset, respectively — matching DeepSeekMath with only 3% of the pretraining tokens.

repo: github.com/microsoft/rho
abs: arxiv.org/abs/2404.07965

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating…

Apple presents Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models

Substantial improvements over SotA VLMs, thanks to its high-resolution scaling and fine-grained visual processing

arxiv.org/abs/2404.07973

LLoCO: Learning Long Contexts Offline

- Significantly outperforms ICL while using 30x fewer tokens during inference
- Achieves up to 7.62x speed-up and substantially reduces the cost of long document QA

arxiv.org/abs/2404.07979

ControlNet++: Improving Conditional Controls
with Efficient Consistency Feedback

Proposes an approach that improves controllable generation by explicitly optimizing pixel-level cycle consistency

proj: liming-ai.github.io/ControlNet_Plu…
abs: arxiv.org/abs/2404.07987

Only true long context language modeing is message-passing via gradient descent (sometimes w/ retrieval)

Attention international students in the U.S. exploring work options!

Did you know that unlike OPT or full-time CPT, part-time CPT (under 20 hours/week) is not subject to a cap?

This means that you can work for as many semesters as you want w/o affecting your future CPT / OPT!