Making diffusion language models work as well as autoregressive ones will be a challenge (see my earlier blog post: sander.ai/2023/01/09/dif…). This paper quantifies this and finds a 64x efficiency disadvantage across all scales 👀 a big gap, but at least it's a constant factor!

PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model This model blends latent semantic diffusion with autoregressive generation. The goal is to produce fluent text while also maintaining global control over paragraph structures. arxiv.org/abs/2306.02531

This paper introduces Language Models Augmented with Long-Term Memory, a novel framework that enhances LLMs capabilities. LongMem uses a decoupled network to handle long-term contexts, greatly improving memory-augmented in-context learning. arxiv.org/abs/2306.07174

1948: Co-locate instructions and data in same memory system 2023: Co-locate instructions and data in the same prompt

The Memory Module is coming. arxiv.org/abs/2307.06945

Raising alert level to Sapphire Sky 2

Raising alert level to Sapphire Sky 3

Raising alert level to Sapphire Sky 4

Probability that AGI has already emerged is now above 1%

The best way to leverage distributed/decentralized compute at scale for AI may turn out to be doing massively parallel inference to generate synthetic data.

Raising alert level to Sapphire Sky 5

Big weakness of equivariant models is that they are leaky abstractions

Raising alert level to Amber Sparkle 1

AGI emerges in less than 1000 days

Raising alert level to Amber Sparkle 2

Probability that AGI has already emerged is now above 2%

Raising alert level to Amber Sparkle 3

Raising alert level to Amber Sparkle 4

Raising alert level to Amber Sparkle 5

Raising alert level to Amber Sparkle 6