@iscienceluvr : Practical Efficiency of Muon for Pretraining "We demonstrate that Muon, the simplest instantiation of a second-order optimizer, explicitly expands the Pareto frontier over AdamW on the compute-time tradeoff. We find that Muon is more effective than AdamW in retaining data • TwiCopy

Tanishq Mathew Abraham, Ph.D.

@iscienceluvr

+ Follow

ID: 441465751

linkhttps://tanishq.ai calendar_today20-12-2011 03:45:50

16,16K Tweet

75,75K Followers

1,1K Following

Tanishq Mathew Abraham, Ph.D.

@iscienceluvr

2 months ago

Practical Efficiency of Muon for Pretraining "We demonstrate that Muon, the simplest instantiation of a second-order optimizer, explicitly expands the Pareto frontier over AdamW on the compute-time tradeoff. We find that Muon is more effective than AdamW in retaining data

thumb_up_off_alt415

chat_bubble_outline10

repeat62

shareShare