🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Transformers can be made sparse across their depth. When trained isoFLOP, we can match or exceed the performance of vanilla models, while saving inference FLOPs arxiv.org/abs/2404.02258

Big fan of the work of Adam Santoro and others. Glad Google decided to finally release it! arxiv.org/abs/2404.02258

... and with one experiment, I was able to roughly reproduce their results for a ~220M GPT-2. It gives a speedup of ~20min (80min dense vs 60min MoD, 4 A100s) while keeping the pplx close. This roughly matches Fig. 3 or 4 in the paper arxiv.org/pdf/2404.02258…

I have implemented Mixture-of-Depths and it shows significant memory reduction during training and 10% speed increase. I will verify if it achieves the same quality with 12.5% active tokens. github.com/thepowerfuldee… thanks Alex Hägele for initial code

A new method for reviews phdcomics.com/comics.php?f=1…