📣With Ian Arawjo (@[email protected]) @tedunderwood.me 🦋 we are studying the “Rise of Preprint Culture in Computing.” We are interested in understanding computing (AI, HCI, etc.) researchers’ practices and perceptions regarding preprints (arXiv, ResearchGate, etc.). You are invited to participate in an…

The first author Yuqi Chen exclusively comes from a history background, yet she led all the challenging computational work in this remarkable paper. A non-STEM background should never deter people from exploring. Or, in Yuqi's own words, 'new technology belongs to everyone.'

It's becoming a very stinky platform.

Since it’s now official in the French press, announcing that the large French open corpus will merge at some point into an European one, code name EuroPile (an obvious reference to EleutherAI). liberation.fr/culture/ia-et-…

Embeddings are at the core of our business model at Nomic AI. That's why we took the time and effort to train the best long-context text embedding model there is, to integrate across our system, and to open source everything about it--code, data, and weights.

Hourglass + Diffusion = ❤️

We introduce a new transformer backbone for diffusion models that can directly generate megapixel images without the need for multiple stages like latent diffusion.

Read here! → arxiv.org/abs/2401.11605
Project page → crowsonkb.github.io/hourglass-diff…

flight

Really excited for this hire--come join Nomic and help us redesign the way datasets, search, servers and browsers fit together to take advantage of what's newly possible with data on the web. nomic-ai.notion.site/Careers-Nomic-…

New post: Have been meaning to write something around what has fundamentally changed around the process of putting ML into prod now that we have LLMs. TL;DR: It's still just compression, we just don't control as much anymore.

vickiboykis.com/2024/01/15/wha…

I learned so much from Lucy Li’s analysis!

Lucy came up with super clever techniques for identifying topics, professions, and location of AboutMe pages (all open sourced!), and discovered correlations w filters used in LLM data selection

AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

Shows that some quality classifiers act like topical domain filters, and langID can overlook English content from some regions of the world

repo: github.com/lucy3/whos_fil……

In Mamba, the selection mechanism has a knob to modulate the flow of time, via Δt. If an input sets Δt → 0, time is effectively frozen, so the state value is momentarily prevented from changing, which acts to 'hold' or 'latch onto' a memory. And Δt → ∞ fast-forwards to reset!