saw someone’s head explode when i played opm babi yesterday

reach out if you want to work with me and others on novel architectures for pretraining! dms are open jobs.ashbyhq.com/zyphra/e509d43…

none of u understand attention sink

A fun fact: Adam remains the dominant optimizer today, yet even it has had only scant opportunities to be verified on trillion-parameter models; Muon, proposed less than a year ago, has already trained at that scale.

the hills of sf are awesome

exit the conversation once it feels too godel escher bach

that tiramisu be sedating me no lie

i set high aura hps for pretrain like a numerologist

bro is living in kowloon walled city monsoon season

I’m nice at ping pong

one time I jumped in a pond exactly like this and stepped on broken glass

guan yin

inference flops don’t matter

new paper arxiv.org/abs/2510.04476

New day, new efficient attention variant

Zyphra is excited to release Compressed Convolutional Attention (CCA), a novel attention mechanism that: - Beats MHA, GQA, MLA for dense and MoE models - Reduces training/prefill flops - 3x fewer parameters vs MHA - Matches GQA/MLA KV-cache sizes without quality penalty

This one was super fun! For the first time we managed to directly reduce training and prefill attention flops directly. CCA is best of both worlds: matching MLA decode speed while substantially outperforming everything in perplexity and prefill speed.

thai iced tea provides me with the power of 10000 suns