Mixture of Experts (MoE)

An architecture that routes each token to a few specialized sub-networks, boosting capacity without proportional cost.

A MoE model may hold hundreds of billions of parameters but activate only a fraction per token, so it runs faster than a dense model of the same size. Mixtral and several frontier models use this design.