Zyphra is releasing our first reasoning model, ZR1-1.5B. This small but powerful reasoning model excels at both math and code, making it one of the best models in these categories for its size. It also uses 60% less reasoning tokens than comparable models. 🆓Apache 2.0 license.

Learning in real time, during deployment, i.e. doing online-continual learning, effectively is important for many applications. It's also associated with theories of intelligence that emphasize learning efficiency, and is an ability where the gap between animals and AI is large.

reach out if you want to work with me and others on novel architectures for pretraining! dms are open jobs.ashbyhq.com/zyphra/e509d43…

It’s not just about GPUs. It’s about the ecosystem. Quentin Anthony joined Jeff Tatarchuk on the Beyond CUDA podcast to share how moving to AMD MI300X cut training costs at Zyphra 📺 Watch the full episode on YouTube (link in comments)

Read more at the blog post here: newsroom.ibm.com/2025-10-01-ibm…

new paper arxiv.org/abs/2510.04476

Zyphra is excited to release Compressed Convolutional Attention (CCA), a novel attention mechanism that: - Beats MHA, GQA, MLA for dense and MoE models - Reduces training/prefill flops - 3x fewer parameters vs MHA - Matches GQA/MLA KV-cache sizes without quality penalty

At this point in attention-free architectures, so many people have poisoned the well that it's just a well of poison. A "Transformer Killer™" drops once a month, and then the authors come back and "kill" transformers again like 5 months later. Love the work, I'm knee-deep in a

Hi Jeff Dean, what’s the plan for releasing the code for this line of work? None of these papers so far seem to have released any code

Today Zyphra releases OVQ-attention, an advancement for efficient long-context processing! Existing LLM layers compress input too much, leading to poor long-context understanding, or too little, leading to expensive memory+compute. OVQ-attention is an alternative path. 🧵

Nice summary.👇

Zyphra is still under the radar but doing truly innovative architecture work

Zyphra releases research on a new way to build hybrid models. We introduce a new architecture leveraging the complementary strengths of Transformers and RNNs for greater flexibility and performance than existing approaches. We call it Hybrid Associative Memory (HAM). 🧵

Leon and Kamesh Krishnamurthy really cooked with this one! arxiv.org/abs/2603.22325

Computer scientists often seem incredibly confident one way or the other about computational functionalism. What they should say is that the arguments both for and against provide only inconclusive considerations and the right attitude is therefore one of great uncertainty.