What does tokens per second mean?
Tokens per second (tok/s) is the rate at which a model generates output — the headline speed number for local inference. Roughly 7-10 tok/s reads as slightly slow, 15-20 is comfortable, and 30+ feels instant.
It is bounded by memory bandwidth, not compute
Generating each token requires streaming the entire set of active weights from memory once. So:
max tok/s ≈ memory_bandwidth (GB/s) / active_model_size (GB)
A 4-bit 70B model is ~40 GB; on a card with 1000 GB/s of bandwidth the ceiling is ~25 tok/s, and real-world numbers land 30-50% below the ceiling. This is why raw TFLOPS is a poor predictor and why unified memory bandwidth is the number to compare across Apple Silicon.
What moves it
| Factor | Effect |
|---|---|
| Quantization | Smaller weights → higher ceiling |
| Batch size | Higher throughput total, similar per-request |
| Prompt length | Longer prompt = slower first token (prefill), then steady |
| Speculative decoding | 1.5-3x with a small draft model |
Prefill (processing your prompt) and decode (generating) are different regimes — a tok/s figure almost always refers to decode.
END OF ANALYSIS