Who published Megatron-LM, and where?

Shoeybi et al. — arXiv 2019 (arXiv:1909.08053).

Where can I find a visual explainer of Megatron-LM?

Right here — a free, interactive, animated walkthrough of the whole paper, with exhibits computed from the real formulas and verbatim quotes from the source.

Megatron-LM

Split a model across GPUs along the matrix — and train billions of parameters.

Shoeybi et al. · arXiv 2019 · Parallelism. Read the paper ↗

A free, interactive, animated visual explainer of Megatron-LM — every exhibit computed from the real formulas, with verbatim quotes from the source.

Questions

What is Megatron-LM?: Split a model across GPUs along the matrix — and train billions of parameters.
Who published Megatron-LM, and where?: Shoeybi et al. — arXiv 2019 (arXiv:1909.08053).
Where can I find a visual explainer of Megatron-LM?: Right here — a free, interactive, animated walkthrough of the whole paper, with exhibits computed from the real formulas and verbatim quotes from the source.

Related explainers

ZeRO: Zero Redundancy Optimizer
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
GSPMD: General and Scalable Parallelization for ML Computation Graphs
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding