ZeRO: Zero Redundancy Optimizer

Partition a model across GPUs instead of replicating it — and train toward a trillion parameters.

Rajbhandari et al. · SC 2020 · Parallelism. Read the paper ↗

A free, interactive, animated visual explainer of ZeRO: Zero Redundancy Optimizer — every exhibit computed from the real formulas, with verbatim quotes from the source.

Questions

What is ZeRO: Zero Redundancy Optimizer?

Partition a model across GPUs instead of replicating it — and train toward a trillion parameters.

Who published ZeRO: Zero Redundancy Optimizer, and where?

Rajbhandari et al. — SC 2020 (arXiv:1910.02054).

Where can I find a visual explainer of ZeRO: Zero Redundancy Optimizer?

Right here — a free, interactive, animated walkthrough of the whole paper, with exhibits computed from the real formulas and verbatim quotes from the source.

Related explainers

• Megatron-LM
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• GSPMD: General and Scalable Parallelization for ML Computation Graphs
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Agent Workflow Memory