Paper2014
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · ICLR · 2014
Abstract
An optimiser that keeps running averages of both the gradient and its square, giving each parameter its own effective step size that adapts as training proceeds, with a bias correction for the early steps when those averages are still cold. It needs little tuning and works across a wide range of problems.
Why it matters
Still the default optimiser for essentially every model on this shelf, more than a decade on. Worth reading precisely because it is the piece of the stack most people never look inside.
trainingoptimizationpretraining
Read the source
https://arxiv.org/abs/1412.6980