Hyped to share JaxPruner: a concise library for sparsity research. JaxPruner includes 10+ easy-to-modify baseline algorithms and provides integration with popular libraries like t5x, scenic, dopamine and fedjax. 1/7 Code: github.com/google-researc… Paper: arxiv.org/pdf/2304.14082…

Why not get the best of both worlds by combining SSMs and Transformers? Excited to share our work at #NeurIPS2023: "Block-State Transformers." BST hits new highs in long-range language modeling and LRA tasks. paper: arxiv.org/abs/2306.09539 1/

Rishi I like the SSM/hyena/Block State Transformers arxiv.org/pdf/2306.09539… arxiv.org/pdf/2302.10866… They remind me of Q-RNNs arxiv.org/abs/1611.01576 and play around with different parallelization ideas. I don't think transformers are that special and there are many equivalent

My research group Krueger AI Safety Lab is looking for interns! Applications are due in 2 weeks ***January 29***. The long-awaited form: forms.gle/iLU1uQAxZ2UKEN… Please share widely!!

Course Correcting Koopman Representations Accepted at #ICLR2024! We identify problems with unrolling in imagination and propose an unconventional, simple, yet effective solution: periodically "𝒓𝒆𝒆𝒏𝒄𝒐𝒅𝒊𝒏𝒈" the latent. 📄 arxiv.org/abs/2310.15386 Google DeepMind 1/🧵

Last week, I gave a talk at Mila - Institut québécois d'IA. The talk should be of interest to anyone working on predictive models, particularly in latent space. In collab. with Mahan Fathi Clement Gehring Jonathan Pilault David Kanaa Pierre-Luc Bacon. See you at ICLR 2026 in 🇦🇹! drive.google.com/file/d/1mQSXFa…

Zyphra is pleased to announce Zamba-7B: - 7B Mamba/Attention hybrid - Competitive with Mistral-7B and Gemma-7B on only 1T fully open training tokens - Outperforms Llama-2 7B and OLMo-7B - All checkpoints across training to be released (Apache 2.0) - Achieved by 7 people, on 128

Zyphra is ecstatic to release Zamba2-small: - 2.7B Mamba2/Attention hybrid - Pre-trained on 3T tokens + annealed on 100B high-quality tokens - Model released on HuggingFace and standalone PyTorch - SOTA evaluation performance and superior inference efficiency.

Yann LeCun Thanks for sharing! Another little trick that might amuse you is that we identified a function which upon minimization produces the forward pass of the attention block:

1) RAG often struggles on complex multi-hop queries. In this blog, we Zyphra discuss and build a graph-based RAG system which tops the leaderboard on a QA benchmark with multi-hop queries and outperforms frontier long-context models for 60x less cost. zyphra.com/post/understan…

I am extremely proud of what the team at Zyphra has achieved. Let's keep pushing the boundaries!

33% of online shoppers in Canada leave their cart before check-out due to high shipping costs (emarketer). Free shipping is must in Canada

#Montreal should be the #innovation and #start-up gate keeper between Europe and the US. Wouldn't entering respective markets be easier?

Its 2012, Canadian eCommerce stuck in the 90's: marketdrum.com/its-2012-canad…

Tips for non-technical entrepreneurs goo.gl/ItgdM