A Mixture of Experts is a neural network architecture that divides its parameters into specialized sub-networks (experts), with a gating mechanism that routes each input to only a subset of relevant experts. This allows the model to have many total parameters while only activating a fraction per input, achieving large-model capability with lower compute cost. Mixtral 8x7B and GPT-4 are believed to use MoE architectures. The trade-off is more complex infrastructure and potential load-balancing challenges.