Expert Parallelism Explainer
Expert parallelism is a distributed-training strategy for mixture-of-experts models. Different expert subnetworks are placed on different devices rather than replicating every expert on every device.
A router assigns each token to one or a small number of experts. Tokens are exchanged across devices, processed by the selected expert networks, and then returned to continue through the model.
Because only a subset of experts is active for each token, the model can have many parameters without using all of them for every example. Expert parallelism supplies the memory and compute distribution needed to scale that structure.
Uneven routing can overload popular experts while others sit idle. All-to-all communication may dominate runtime, and capacity limits can drop or reroute tokens. Training needs load-balancing objectives, careful placement, adequate network bandwidth, and quality checks by domain.
Expert parallelism distributes the expert subnetworks of a mixture-of-experts model across devices and routes tokens to the selected experts.
Key terms include expert parallelism, mixture of experts, router, token, expert network, all-to-all communication, capacity, load balancing and device.
The router selects a small set of experts for each token.
Distributed communication sends tokens to the devices holding those experts and returns their outputs.
Popular experts can exceed capacity while underused experts waste available computation.
Load balancing and fast all-to-all links are central to practical performance.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.