"New State of the Art AI Optimizer: Rectified Adam (RAdam). Improve your AI accuracy instantly versus Adam, & why it works" It's been a long time since we've seen a new optimizer reliably beat the old favorites; this looks like a very encouraging approach! link.medium.com/iN3d4LMpbZ