Mistral Large 4 in open weights: a 1T MoE (52B active) for on-premise inference

Mistral just released Mistral Large 4 in open weights: 1 trillion total parameters, 52 billion active per token, a 1M token context. On paper it is the largest model ever made available in open weights. In practice, the right question is not "how strong is it" but "what does it fit on, at what quantization, and what speed do you get". This article does the math, because a 1T MoE reads very differently from a 1T dense.
MoE in practice: what "1T total, 52B active" means
If you have already read how the Mixture-of-Experts works, the concept is simple: each token is processed by a subset of the weights, not all of them. The 1T total parameters determine what the model knows (capacity, which grows with the memory it occupies). The 52B active determine what each token costs (speed, which depends on memory bandwidth).
The practical consequence: quality scales roughly like a large dense model, but speed stays at medium-model levels. That is the trick that Aleph Alpha's Kolibri (78B total, 12B active) and Nemotron 3.5 Bolt (74B total, ~9B active) already proved at smaller scale. Mistral Large 4 scales the same scheme up to 1T.
Specs from Mistral's model card:
| Parameter | Value |
|---|---|
| Architecture | Mixed MoE (dense + MoE layers), 128 experts, 4 active/token + 1 shared |
| Total parameters | 1.03T |
| Active parameters | 52B |
| Context window | 256K default, 1M max |
| Input | text + images |
| License | Mistral Ecosystem Open-Weight (modified MIT) |
| Native quantization | FP8 (source checkpoint) |
The Mistral Ecosystem Open-Weight license matters: it is a modified MIT, commercial use is free — that is the license teams look for when the model ends up behind an internal product. A separate commercial license is required for a Model-as-a-Service business.
The math: what it takes to run it
Here are the numbers that decide. Weights occupy memory, and each quantization is a quality/space trade-off. Official sizes from ggml-org's GGUFs:
| Quantization | Size | Fits in |
|---|---|---|
| BF16 | 2.10 TB | multi-GPU servers with unified memory |
| Q8_0 | 1.12 TB | near-lossless |
| Q4_K_M | 635 GB | cluster with ~1 TB of RAM |
| Q2_K | 381 GB | Mac Studio M5 Ultra 512GB (with reduced KV cache) |
| IQ2_XXS | 298 GB | Mac Studio M5 Ultra 512GB |
On top of that comes the KV cache, which grows with the context you use: with 256K context tokens and 52B active parameters, the cache alone can take dozens of GB. If you want full context, the budget has to be computed on the total (weights + cache), not on weights alone.
The honest read: IQ2_XXS at 298 GB is the only quantization that fits on a single consumer machine (the Mac Studio M5 Ultra with 512GB) with room for the cache. Q4_K_M at 635 GB requires a system with at least 1 TB of unified RAM — for example a multi-node cluster with RDMA, or two 512GB Mac Studios in a multi-machine setup, which adds latency.
What speed to expect
The rule for MoE models: decode speed depends on the active weights, prefill speed on the total weights. Prefill reads all 1T parameters (or whatever quantization you use), so long contexts cost more than on a dense model. Decode reads only ~52B active weights per token, so it stays reasonable.
The math for the consumer machine that fits, the Mac Studio M5 Ultra with 512GB and ~819 GB/s of unified memory bandwidth:
| Scenario | Weights read/token | Theoretical estimate |
|---|---|---|
| Q2_K (381 GB), decode | ~52B active → ~29 GB | ~819 / 29 ≈ 28 tok/s |
| IQ2_XXS (298 GB), decode | ~52B active → ~25 GB | ~819 / 25 ≈ 32 tok/s |
These are bandwidth estimates (active weights ÷ memory bandwidth), not benchmarks: in practice expect 20-30% less, for attention, cache and expert routing. Prefill at 1T is much slower than decode: a request with 100K context tokens has a prefill cost that would be prohibitive in RAM on a 1T dense, but here the weights fit in 298-381 GB quantized.
If instead you target a 1 TB RAM system with Q4_K_M, quality jumps but decode drops: more active weights to load per token. The quantization choice is a quality/speed trade-off depending on what you need: for a conversational assistant with moderate context, Q2_K or IQ2_XXS on an M5 Ultra are already usable. For a RAG with long contexts and many simultaneous users, you need the 635 GB of Q4_K_M and multi-machine hardware.
Why a 1T open-weight model changes the on-premise discussion
Until yesterday, a model of this size in open weights did not exist: Llama 3.1's 405B was the ceiling, and it already required 8 GPUs with 80GB. With 1T in open weights, a cluster that last week looked like training infrastructure becomes inference infrastructure for real products. The practical difference: you put a frontier-class model behind your product, your data stays in your perimeter, and costs do not grow with the number of users — that is the break-even point we computed in costs: local vs API.
For a business, this opens a concrete use case: the assistant over internal documents with contexts of hundreds of thousands of tokens, which on a public API costs per request. On-premise, the cost is the infrastructure you already have, and privacy is by construction, not by contract.
How to get started
- Check the hardware: you need at least 320 GB of unified RAM for IQ2_XXS. The Mac Studio M5 Ultra with 512GB is today the only single-chassis consumer machine that fits it with room for the KV cache.
- Install llama.cpp:
curl -LsSf https://llama.app/install.sh | sh
- Start the server with the quantization that fits your hardware:
llama serve -hf ggml-org/Mistral-Large-4-1T-A52B-GGUF:Q2_K
- Test with a prompt: open the web UI that
llama serveexposes, or:
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"default","messages":[{"role":"user","content":"Hello"}]}'
If it does not work: if the server fails to start, the problem is almost always RAM — check that your chosen quantization plus the estimated KV cache does not exceed the RAM available. With reasoning_effort at high (the default), answers are more thorough but slower: for quick tests, set it to low.
The limits, to be honest
- Contexts cost: 1M context tokens is a capability, not a habit. Full-context KV cache requires dozens of extra GB, and prefill on long inputs is slow.
- It is not for a single light user: if you have short chats on a laptop, this model is overkill. For that case, an 8-32B model does everything you need.
- Speed is not 4B-class: 52B active weights per token give good speeds but not instant ones. If latency is critical, a smaller model remains the right choice.
Sources
In summary, Mistral Large 4 brings one trillion parameters to open weights under the Mistral Ecosystem Open-Weight license (a modified MIT). The MoE with 52B active parameters makes it runnable at reasonable decode speeds: IQ2_XXS at 298 GB fits a Mac Studio M5 Ultra 512GB, Q4_K_M at 635 GB opens the door to 1 TB clusters. The context up to 1M tokens and image input place it in the frontier class for on-premise workloads. If your use case has long contexts and sensitive data, it is the first open-weight model worth putting on your list.
Article generated with AI
In these reading paths
Enjoyed this article?
Subscribe to the newsletter for new tutorials and guides.
Related Articles

Qwen3.8-2.4T-A95B: the math of running 2.4T open-weight parameters, 95B active
The first Qwen-Max-class open-weight model weighs ~1.3 TB at Q4: no single consumer node fits it. The math on amortization, power, and speed for multi-node.

Ripple-65B: a 64.8B dense in Apache-2.0, the math of running it locally
64.8B parameters, Apache 2.0, 1M of context and an NVFP4 checkpoint designed for 128 GB of unified memory: the math and the criteria for running a frontier dense locally.

