Interactive Utility
Mixtral 8x7B VRAM Calculator
Stop guessing your hardware needs for Mixtral 8x7B. Calculate the exact VRAM required for inference based on parameter count, quantization, and KV cache, and see which machines can actually run it.
Model
Custom: billion
The community default. Best size-to-quality ratio.
Memory to run it
- Weights (Q4_K_M)
- 28.5 GB
- KV cache (8k)
- 3.4 GB
- Runtime overhead
- 1.5 GB
- System RAM to load
- 30 GB
- Model file on disk
- 29.9 GB
Hardware match
2× RTX 3090 / 4090, or Mac Studio (64GB+)
Multi-GPU splitting or unified memory. Expect a PCIe throughput penalty on the dual-GPU path.
Rough decode speed
~19 tok/s
Ballpark on 2×4090 / RTX 6000, bounded by memory bandwidth. Real numbers land 30–50% either side depending on framework, prompt length and thermals.
Mixtral 8x7B memory requirements by quantisation
Estimated at an 8k-token context. Mixtral 8x7B is 47B parameters; the arithmetic and its assumptions are set out below.
| Quantisation | Weights | Total VRAM | Runs on |
|---|---|---|---|
| FP16 (unquantised) | 94.0 GB | 98.9 GB | Mac Studio Ultra (128GB) or an 80GB data-center GPU |
| Q8_0 (8-bit) | 49.9 GB | 54.8 GB | Mac Studio Ultra (128GB) or an 80GB data-center GPU |
| Q6_K (6-bit) | 38.8 GB | 43.6 GB | 2× RTX 3090 / 4090, or Mac Studio (64GB+) |
| Q5_K_M (5-bit) | 33.5 GB | 38.3 GB | 2× RTX 3090 / 4090, or Mac Studio (64GB+) |
| Q4_K_M (4-bit) | 28.5 GB | 33.4 GB | 2× RTX 3090 / 4090, or Mac Studio (64GB+) |
| Q3_K_M (3-bit) | 22.9 GB | 27.8 GB | 2× RTX 3090 / 4090, or Mac Studio (64GB+) |
Arithmetic over stated assumptions, not a benchmark. Weights are parameter count times bits-per-weight; the KV cache term assumes grouped-query attention; 1.5GB is reserved for runtime overhead.
Size a related model
Embed this Mixtral 8x7B Calculator
Add this exact configuration to your blog, documentation, or internal wiki.
<iframe src="https://fathomlayer.com/embed/hardware-calculator?p=47" width="100%" height="600" frameborder="0" style="border-radius: 12px; border: 1px solid rgba(255,255,255,0.1);"></iframe>
How this math works
Running Large Language Models locally is entirely bottlenecked by memory. Raw compute (TFLOPS) dictates your generation speed, but VRAM capacity dictates if the model will load at all.
- Model Weights: At FP16 (unquantized), every 1 Billion parameters requires ~2GB of VRAM. At Q4 (4-bit quantization), that drops to ~0.7GB per 1B parameters.
- KV Cache (Context Window): As you feed text into the model, it stores attention states in memory. A 32k context window on a 70B model requires several extra Gigabytes of RAM independent of the model weights.
- Overhead: CUDA and operating systems reserve memory (usually 1-2GB), meaning a 24GB RTX 4090 cannot realistically load a 23.5GB model.