GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

One dial from MQA to MHA — near-MHA quality at near-MQA decode speed, retrofitted cheaply.

Ainslie et al. · EMNLP 2023 · Model Architectures. Read the paper ↗

A free, interactive, animated visual explainer of GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — every exhibit computed from the real formulas, with verbatim quotes from the source.

Questions

What is GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints?

One dial from MQA to MHA — near-MHA quality at near-MQA decode speed, retrofitted cheaply.

Who published GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, and where?

Ainslie et al. — EMNLP 2023 (arXiv:2305.13245).

Where can I find a visual explainer of GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints?

Right here — a free, interactive, animated walkthrough of the whole paper, with exhibits computed from the real formulas and verbatim quotes from the source.

Related explainers

• PagedAttention (vLLM)
• DeepSeek-V3
• Qwen3
• OLMo 2
• MiniMax-01
• Gemma 4
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality