@peterjliu : We recently open-sourced a relatively minimal implementation example of Transformer language model training in JAX, called NanoDO. If you stick to vanilla JAX components, the code is relatively straightforward to read -- the model file is <1550 lines. We found it useful as a • TwiCopy

Peter J. Liu

@peterjliu

+ Follow

AI research-eneur. Was Research Scientist @ Google Brain / DeepMind, language model research.

ID: 41280228

linkhttp://peterjliu.com calendar_today20-05-2009 02:51:10

683 Tweet

8,8K Takipçi

2,2K Takip Edilen

Peter J. Liu

@peterjliu

a year ago

We recently open-sourced a relatively minimal implementation example of Transformer language model training in JAX, called NanoDO. If you stick to vanilla JAX components, the code is relatively straightforward to read -- the model file is <150 lines. We found it useful as a

thumb_up_off_alt278

chat_bubble_outline3

repeat60

shareShare