Mixture-of-Experts (MoE)

Model architecture in which the model counts many billions of parameters in total, but for each token activates only a small subset (the 'experts' selected by a router). The result is the quality of a large model with the speed and compute cost of a small one. Example: Qwen3 30B-A3B has 30.5B total parameters but uses only 3.3B per token. The price is that the whole model weight must still be loaded into memory.