Mihir Patel (@mvpatel2000) 's Twitter Profile
Mihir Patel

@mvpatel2000

Member of Technical Staff @AnthropicAI. prev @MosaicML/@DbrxMosaicAI, @Stanford. Views do not represent my employer.

ID: 1324081883173498882

calendar_today04-11-2020 20:11:42

1,1K Tweet

5,5K Takipçi

481 Takip Edilen

Mihir Patel (@mvpatel2000) 's Twitter Profile Photo
Mihir Patel
@mvpatel2000

a year ago

Gemma 2 is out with 9b and 27b! A few things I really liked in tech report: On pretraining: - GQA (finally lol) - interleaving global attn w local (but 4k vs. 8k? it feels like it should support longer and was kneecapped...) - 16x expansion ratio (very wide!) (1/n)

Gemma 2 is out with 9b and 27b! A few things I really liked in tech report: On pretraining: - GQA (finally lol) - interleaving global attn w local (but 4k vs. 8k? it feels like it should support longer and was kneecapped...) - 16x expansion ratio (very wide!) (1/n)