Code Llama (70B)
VRAM calculator
One model, different memory footprints. Compare weight formats, then explore the context and memory budget that matter to your setup.
The same model.
Six ways to hold its weights.
8k-token context
24 GB usable memory entered
Estimate, not a guarantee of fit or performance.
| Weight format | Weights | KV cache | Runtime | Total | 24 GB budget | Open format scenario |
|---|---|---|---|---|---|---|
| FP16 (unquantised)16 effective bits / weight | 140.0 | 3.4 | 1.5 | 144.9 | 120.9shortfall | Use format |
| Q8_0 (8-bit)8.5 effective bits / weight | 74.4 | 3.4 | 1.5 | 79.2 | 55.2shortfall | Use format |
| Q6_K (6-bit)6.6 effective bits / weight | 57.8 | 3.4 | 1.5 | 62.6 | 38.6shortfall | Use format |
| Q5_K_M (5-bit)5.7 effective bits / weight | 49.9 | 3.4 | 1.5 | 54.7 | 30.7shortfall | Use format |
| Q4_K_M (4-bit)4.85 effective bits / weight · selected | 42.4 | 3.4 | 1.5 | 47.3 | 23.3shortfall | Use format |
| Q3_K_M (3-bit)3.9 effective bits / weight | 34.1 | 3.4 | 1.5 | 39.0 | 15.0shortfall | Use format |
This table shows the submitted scenario. Live edits preview in the calculator below; Calculate scenario updates both the table and calculator. Arithmetic over stated assumptions, not measurements. Cache depends on architecture and precision; weight format alone does not establish model quality. The table and reference answer both use an 8k-token context.
Change the budget.
Keep the model in view.
70B parameters are fixed on this page.
Use a custom model size ↗
Estimated device memory
- Model weightsQ4_K_M (4-bit) · 4.85 effective bits / weight
- 42.4 GB
- KV cache8k tokens · generic FP16 cache estimate
- 3.4 GB
- Runtime allowanceFixed allowance in this planning model
- 1.5 GB
Weights exceed the memory entered
The estimate exceeds your usable memory. Try a smaller model, a different weight format or a shorter context, and check the runtime requirements.
- System RAM to load estimate
- 44 GB
- Model file size estimate
- 44.6 GB
Arithmetic over stated assumptions, not a benchmark. Architecture, KV heads, cache precision, runtime and offloading can change the actual footprint.
A useful estimate
has visible limits.
How much VRAM does Code Llama (70B) need?
Code Llama (70B) (70B parameters) needs about 47.3 GB of VRAM at Q4_K_M and about 144.9 GB at FP16, with an 8k-token context. Estimate from parameter count, bits per weight, KV cache and 1.5 GB of runtime overhead — not a benchmark.
These calculations use the catalog's total parameter count: 70B. For mixture-of-experts models, the total counts all expert weights rather than only the parameters active for a token.
- Model weights
- Parameters × effective bits per weight ÷ 8. Effective bit values include format overhead and represent planning assumptions.
- KV cache
- A generic FP16 cache estimate from parameter-count bands and context length. This is not a measurement of this model's KV-head configuration.
- Runtime allowance
- 1.5 GB is reserved for runtime overhead. Actual framework allocations, offloading and device workloads can change the requirement.
Memory capacity helps determine whether a scenario fits. Memory bandwidth, compute, architecture and runtime also affect speed. This calculator does not predict tokens per second.
Open compact calculator