Differential Transformer is a new foundation architecture for LLMs that enhances focus on relevant information while canceling attention noise. It generally outperforms Transformer models w/advantages in long-sequence modeling & hallucination mitigation. msft.it/6010mhpr6

Microsoft researchers release bitnet.cpp, the official inference framework for 1-bit LLMs like BitNet b1.58. It has optimized kernels for fast, lossless inference on CPUs, achieving impressive speedups on ARM and x86 CPUs and significant energy reductions. msft.it/6016WGy8o

this Google DeepMind paper explores the limitations of the softmax function in artificial intelligence systems, particularly its inability to maintain sharpness in decision-making as the number of inputs increases. key findings the softmax function cannot robustly approximate

Cache smarter, not harder: one-time storage trick makes LLMs fly with less memory. 💡 "You Only Cache Once (YOCO) from Microsoft : Decoder-Decoder Architectures for Language Models" ✨ One cache to rule them all: YOCO keeps LLM memory lean by storing key-value pairs just once.

Multimodal Latent Language Modeling with Next-Token Diffusion From MS Research. Employs a variational autoencoder to represent continuous data as latent vectors and introduces next-token diffusion for autoregressive generation of these vectors. Effective across various

LatentLM makes multimodal AI speak one language by treating all data types as continuous tokens. Next-token diffusion: The universal translator between discrete and continuous data in AI models. LatentLM unifies continuous and discrete data processing in LLMs using causal

Long paper thread! I've held a skeptical opinion of DiffTransformer after skimming it, but after some recent interaction thought that's unfair and I'd give it a proper read. Did so on a recent flight. Paper TL;DR: pair two attention heads, and do: (sm(Q1K1) - λ sm(Q2K2)) V

Forget just thinking in words. 🚀 New Era of Multimodal Reasoning🚨 🔍 Imagine While Reasoning in Space with MVoT Multimodal Visualization-of-Thought (MVoT) revolutionizes reasoning by generating visual "thoughts" that transform how AI thinks, reasons, and explains itself.

GPT-4o's image gen quality, especially with text ✍️, is impressive! But how does it work? Beyond standard diffusion (like Stable Diffusion/DiT), could other architectures be key? Let's explore potential inspiration from recent research, starting with LatentLM. 🧐 #AI #GPT4o (1/6)

🚨 We have updated our ICLR 2025 Oral paper Differential Transformer with additional results on mathematical reasoning. DIFF consistently outperforms Transformer on o1-style reasoning evaluation, achieving an average accuracy improvement of 7.5% (see Fig. 2). (1/3)

Microsoft presents Reward Reasoning Model

Think Only When You Need with Large Hybrid-Reasoning Models

⏰ We introduce Reinforcement Pre-Training (RPT🍒) — reframing next-token prediction as a reasoning task using RLVR ✅ General-purpose reasoning 📑 Scalable RL on web corpus 📈 Stronger pre-training + RLVR results 🚀 Allow allocate more compute on specific tokens

what a figure

New training paradigm: instead of just predicting tokens, models reason about each prediction using RL The model thinks through context, considers alternatives, then makes a prediction. 14B model matches 32B baseline, though training costs are significantly higher.

Reinforcement Pre-Training New pre-training paradigm for LLMs just landed on arXiv! It incentivises effective next-token reasoning with RL. This unlocks richer reasoning capabilities using only raw text and intrinsic RL signals. A must-read! Bookmark it! Here are my notes:

Microsoft just dropped VibeVoice on Hugging Face A novel framework generating expressive, long-form, multi-speaker conversational audio like podcasts from text. Synthesizes up to 90 minutes of speech with up to 4 distinct speakers! huggingface.co/microsoft/Vibe…

Microsoft just released VibeVoice - 1.5B SoTA Text to Speech model - MIT Licensed 🔥 > It can generate up 90 minutes of audio > Supports simultaneous generation of > 4 speakers > Streaming and larger 7B model in-coming > Capable of cross-lingual and singing synthesis Love the

VibeVoice A Frontier Open-Source Text-to-Speech Model

VibeVoice ❤️ Wan huggingface.co/microsoft/Vibe…