Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Abstract

Simplifies mixture-of-experts routing to its limit by sending each token to exactly one expert rather than several, which cuts routing overhead and communication enough to make the approach practical at scale. Parameter count grows enormously while the compute spent per token stays fixed, along with the recipes needed to keep training stable.

https://arxiv.org/abs/2101.03961