GSPMD: General and Scalable Parallelization for ML Computation Graphs

Annotate a few tensors; the compiler shards the trillion-parameter rest.

Xu et al. · arXiv 2021 · Parallelism. Read the paper ↗

A free, interactive, animated visual explainer of GSPMD: General and Scalable Parallelization for ML Computation Graphs — every exhibit computed from the real formulas, with verbatim quotes from the source.

Questions

What is GSPMD: General and Scalable Parallelization for ML Computation Graphs?

Annotate a few tensors; the compiler shards the trillion-parameter rest.

Who published GSPMD: General and Scalable Parallelization for ML Computation Graphs, and where?

Xu et al. — arXiv 2021 (arXiv:2105.04663).

Where can I find a visual explainer of GSPMD: General and Scalable Parallelization for ML Computation Graphs?

Right here — a free, interactive, animated walkthrough of the whole paper, with exhibits computed from the real formulas and verbatim quotes from the source.

Related explainers

• Megatron-LM
• ZeRO: Zero Redundancy Optimizer
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning