GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Top-2 experts per token + an SPMD compiler: a 600B model trained in 4 days.
Lepikhin et al. · ICLR 2021 · Parallelism. Read the paper ↗
A free, interactive, animated visual explainer of GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — every exhibit computed from the real formulas, with verbatim quotes from the source.
Questions
- What is GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding?
- Top-2 experts per token + an SPMD compiler: a 600B model trained in 4 days.
- Who published GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, and where?
- Lepikhin et al. — ICLR 2021 (arXiv:2006.16668).
- Where can I find a visual explainer of GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding?
- Right here — a free, interactive, animated walkthrough of the whole paper, with exhibits computed from the real formulas and verbatim quotes from the source.