🌟Several dataset releases deserve a mention for their incredible data measurement work 🌟

➡️ The Pile (arxiv.org/abs/2101.00027) Leo Gao Stella Biderman

➡️ ROOTS (arxiv.org/abs/2303.03915) Hugo Laurençon++

➡️ Dolma (arxiv.org/abs/2402.00159) Luca Soldaini 🎀 Kyle Lo

14/

this is the one🍪

follow-up to our work on BooookScore:

🐋prev, we evaluated summary coherency,
🦉now, we're evaluating faithfulness, omissions, etc which is hard cuz it requires localizing summary generations within original source (>100k tokens)

come chat w us at ICLR 2024 🐙

PS: if you are also attending GenLaw and are looking for opportunities to research at the intersection of AI, Law, and Policy, let's chat 😊

It’s finally here 🎉🥳

In case you missed us, MosaicML/ Databricks is back at it, with a new best in class open weight LLM named DBRX. An MoE with 132B total parameters and 32B active 32k context length and trained for 12T tokens 🤯

truly cursed timeline 😵‍💫

ACLRollingReview reviews due Mar 20
Conference on Language Modeling abstract deadline Mar 22
ACLRollingReview reviews released Mar 26
Conference on Language Modeling submission deadline Mar 29
ACLRollingReview rebuttal period closes Mar 30

one of my favorite aspects of this project is that it shows careful reuse of high quality 'older' datasets is still effective today🦉

we may think 'instructions' are relatively recent trend in NLP but some of the datasets we repurpose date back to 2004! 🎂

LMs can generate plain language summaries. For some audiences, automated simplification of complex text can improve the reading experience.
But what of users with more subject matter expertise? Our #CHI2024 paper studies benefits & pitfalls of LMs for simplifying science texts.

can LMs help us write expository answers to scientific research questions?

excited to share our work led by Fangyuan Xu. we recruited NLP folks to work with an LM to answer research questions and logged successes/failures in sustained interaction traces🦉

New Resource: Foundation Model Development Cheatsheet for best practices

We compiled 250+ resources & tools for:
🔭 sourcing data
🔍 documenting & audits
🌴 environmental impact
☢️ risks & harms eval
🌍 release & monitoring

With experts from EleutherAI, Allen Institute for AI,…

for those looking for instruction-tuned OLMo👇

DM me if you're interested in:
🐋creating high-quality pretraining datasets
🐊studying data's impact on LM capabilities
🦉tools for sensemaking over large corpora
🐡adapting LMs to specialized domains like science
🐈evaluation through human interaction

ngl anonymous.4open.science is probably the single most useful research tool i've adopted in recent memory

this work is fascinating 🤯
1. they successfully inflate citation counts by creating chatGPT generated articles. google scholar parses the fake article's references & automatically increments citation counts
2. they successfully pay services to literally 'buy' citations

We just uploaded detailed Weights & Biases training logs for the OLMo 7B run: wandb.ai/ai2-llm/OLMo-7…

This is a cleaned-up version from the actual run, so the wall clock times don't make sense, but all the other information is there!

excited to share our contribution to open science of language models!

🐈‍⬛ all our data, weights, ckpts, code, etc
🐈 covers data curation, pretraining, adaptation, evaluation, etc

check out more deets in Luca Soldaini 🎀 ‘s thread, technical reports out on arXiv shortly 😆