Adam: A Method for Stochastic Optimization

A per-parameter adaptive learning rate from two moving averages of the gradient.

Kingma & Ba · ICLR 2015 · Foundations. Read the paper ↗

A free, interactive, animated visual explainer of Adam: A Method for Stochastic Optimization — every exhibit computed from the real formulas, with verbatim quotes from the source.

Questions

What is Adam: A Method for Stochastic Optimization?

A per-parameter adaptive learning rate from two moving averages of the gradient.

Who published Adam: A Method for Stochastic Optimization, and where?

Kingma & Ba — ICLR 2015 (arXiv:1412.6980).

Where can I find a visual explainer of Adam: A Method for Stochastic Optimization?

Right here — a free, interactive, animated walkthrough of the whole paper, with exhibits computed from the real formulas and verbatim quotes from the source.

Related explainers

• Attention Is All You Need
• GPT-3: Language Models are Few-Shot Learners
• Mixtral of Experts
• Training Compute-Optimal Large Language Models
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• BERT: Pre-training of Deep Bidirectional Transformers
• Scaling Laws for Neural Language Models
• Deep Residual Learning for Image Recognition