A Mixture of Experts (MoE) is a neural network architecture that scales parameters efficiently by activating only a small subset of “expert” sub-networks for each input token. A learned router (gating network) selects the top-k experts per token, enabling high capacity with roughly constant compute per token compared to a dense model of similar parameter count.
What is Mixture of Experts (MoE)?
MoE layers replace or augment dense feed-forward blocks with a pool of experts and a router that dispatches tokens to the top-1 or top-2 experts. Each expert processes only its assigned tokens, and outputs are combined via gated weights. Auxiliary load-balancing losses (and capacity factors) prevent expert collapse and overflow. MoE layers are interleaved with attention and can be sparsely activated during both training and inference. Practical systems rely on all-to-all communications across devices, expert sharding, micro-batching, and kernel optimizations to minimize routing overhead. Compared to dense LLMs, MoE achieves better quality-per-compute and specialization at the cost of distributed systems complexity.
FAQs
- How is MoE different from a dense LLM? Only a few experts run per token, so total activated parameters per token are small, even though total parameters are large.
- Does MoE reduce inference cost? Yes, compute per token stays near constant, but memory footprint and interconnect bandwidth needs increase.
- What is top-1 vs top-2 routing? The router sends each token to one or two experts; top-2 tends to improve quality with modest extra compute.
- How do you fine-tune MoE models? Options include tuning the router, specific experts, or using adapter/LoRA layers per expert to control cost.
- What are common pitfalls? Expert imbalance, router instability, token drops at capacity limits, and communication bottlenecks without fast interconnects (e.g., NVLink/InfiniBand).
Why it matters and where it’s used
MoE delivers high model capacity and specialization at a lower amortized compute cost per token, making it attractive for large-scale LLMs, multilingual models, and code or domain-specialized assistants. It enables scaling beyond single-node memory limits but requires sophisticated distributed training and serving stacks with careful routing, caching, and autoscaling policies.
Examples
- Switch Transformer for translation tasks using top-1 routing with auxiliary balancing.
- Mixtral-style 8×7B top-2 MoE that rivals larger dense models in quality at lower inference FLOPs.
- Domain experts: a code-focused expert, a multilingual expert, and a math expert activated selectively by the router for relevant tokens.