Optimising local LLM inference: where the bottleneck actually is
Local inference is memory-bandwidth bound, not compute bound. Generating one token requires streaming every active weight from memory exactly once. So the ceiling is:
max tokens/sec ≈ memory_bandwidth (GB/s) / active_model_size (GB)
A 4-bit 70B model is ~40 GB. On a card with 1000 GB/s of bandwidth the ceiling is ~25 tok/s; real numbers land 30–50% below that. Raw TFLOPS barely moves this — which is why a 5-year-old 3090 and a current 4090 generate 70B tokens at similar speed, and why unified memory bandwidth is the number to compare across Macs.
The levers, in order of impact
| Lever | Typical gain | Cost |
|---|---|---|
| Quantise weights (FP16 → Q4_K_M) | 2–3× | Small quality loss |
| Quantise the KV cache (FP16 → FP8) | Fits longer context; slight speed gain | Minor loss at very long context |
| Speculative decoding (small draft model) | 1.5–3× | Extra VRAM for the draft model |
| Right serving engine (llama.cpp → vLLM/ExLlamaV2) | 1.5–2× at batch > 1 | Setup complexity |
| Flash-attention / paged attention | Bigger batch, less fragmentation | Needs supported hardware |
| Tensor parallelism across 2 GPUs | More VRAM, not more speed | PCIe becomes the bottleneck |
What does not help
- A faster CPU — irrelevant once the model is on the GPU.
- More PCIe lanes — only matters for the initial load and for multi-GPU, and even then NVLink beats PCIe by a wide margin.
- Overclocking the core — you are waiting on memory, not the ALUs. Overclock the memory if anything.
A worked example
Llama 3 70B, target 15 tok/s, 8k context:
- Q4_K_M → ~40 GB weights. Won't fit one 24 GB card.
- Options: 2× 24 GB (48 GB, ~10–14 tok/s after PCIe penalty) or a 64 GB Mac Studio (~14–18 tok/s).
- Add FP8 KV cache → 8k context costs ~1.4 GB instead of ~2.8 GB.
- On the dual-GPU path, use ExLlamaV2 with tensor parallelism; on the Mac, MLX.
The rule
Fix quantisation first (Q4_K_M unless you have a measured reason not to), then match memory bandwidth to your target tokens/sec, then pick the serving engine. Everything else is a rounding error.
END OF ANALYSIS