@gneubig : Summary of GPT-OSS architectural innovations: 1. sliding window attention (ref: arxiv.org/abs/19501.02860) 2. mixture of experts (ref: arxiv.org/abs/2101.03961) 3. RoPE w/ Yarn (ref: arxiv.org/abs/2309.00071) 4. attention sinks (ref: streaming llm arxiv.org/abs/2309.17453) • TwiCopy

Graham Neubig

@gneubig

+ Follow

Associate professor at CMU, studying natural language processing and machine learning. Co-founder @allhands_ai

ID: 185910194

linkhttp://www.phontron.com calendar_today02-09-2010 03:25:38

3,3K Tweet

37,37K Takipçi

668 Takip Edilen

Graham Neubig

@gneubig

3 days ago

Summary of GPT-OSS architectural innovations: 1. sliding window attention (ref: arxiv.org/abs/1901.02860) 2. mixture of experts (ref: arxiv.org/abs/2101.03961) 3. RoPE w/ Yarn (ref: arxiv.org/abs/2309.00071) 4. attention sinks (ref: streaming llm arxiv.org/abs/2309.17453)

thumb_up_off_alt1,1K

chat_bubble_outline9

repeat274

shareShare