Skip to content
Fathom Layer
GlossaryGlossary7 MIN READ

What does tokens per second mean?

Laptop displaying conceptual output blocks and a cursor without performance figures
AI-generated editorial illustration · Conceptual scene

Tokens per second (tok/s) is the rate at which a language model processes or produces tokens, and in our reading, a single tok/s figure quoted for a local model usually means generation speed: how fast new tokens appear after the first one. A token is a piece of text, not a word; NVIDIA puts one token at approximately four English characters. Because a model handles your prompt and its reply in two very different ways, a useful speed report has at least two numbers, and the one that governs how fast text appears is limited mainly by memory bandwidth, not raw compute.

How it works: prefill and decode

NVIDIA describes LLM inference as two phases.

  • Prefill processes the input prompt. All prompt tokens are known in advance, so the work runs in parallel; the vLLM paper describes it as matrix-matrix multiplication, which keeps the GPU busy.
  • Decode generates the reply one token at a time, each depending on the ones before. NVIDIA calls this "like a matrix-vector operation that underutilizes the GPU compute ability", and says the speed of moving weights, keys, values and activations from memory "dominates the latency, not how fast the computation actually happens. In other words, this is a memory-bound operation."

llama.cpp's benchmark tool reflects the split: llama-bench reports prompt processing (pp) and text generation (tg) as separate tests. The two phases are limited by different things, so a tok/s number without its phase is incomplete.

The metrics that matter

NVIDIA's benchmarking documentation defines the terms used by most serving tools:

Metric What it measures
Time to first token (TTFT) Time from submitting a request to receiving the first token; generally includes queueing, prefill and network latency
Inter-token latency (ITL), which NVIDIA says is "also known as time per output token (TPOT)" Average time between consecutive generated tokens; NVIDIA notes tools differ on whether the first token is included, and its AIPerf excludes TTFT
TPS per user Output length divided by end-to-end latency for one request; approaches 1/ITL as outputs get longer
Total TPS per system Output tokens across all simultaneous requests divided by the benchmark window
End-to-end latency TTFT plus generation time

NVIDIA notes that total TPS rises as more requests are added until it saturates the GPU, and can fall after that. A server can therefore report a high aggregate tok/s while each user sees modest speed. A local chat app with one user reports something closer to TPS per user.

NVIDIA also notes that longer prompts increase TTFT, because the full input has to be processed to build the KV cache before generation begins.

Why memory bandwidth caps decode speed

For a dense model generating for one user, each new token requires reading the model's weights from memory. That gives a rough ceiling, which is our arithmetic, not a vendor figure or a measurement:

max decode tok/s ≈ memory bandwidth (bytes/s) ÷ bytes read per token
bytes read per token ≈ parameters × bytes per parameter (+ the KV cache for the context so far)

Worked through for an 8-billion-parameter model at 16-bit (8 × 10⁹ × 2 bytes = 16 GB of weights), using manufacturers' published bandwidth:

Hardware Published bandwidth Ceiling for 16 GB of weights
NVIDIA GeForce RTX 5090 1,792 GB/sec about 112 tok/s
Apple Mac Studio, M5 Ultra 1.2 TB/s about 75 tok/s
NVIDIA DGX Spark 273 GB/s about 17 tok/s

These are upper bounds. Real throughput is lower because kernels, software overhead and the KV cache also consume time and bandwidth, and this index has not measured these systems. The point is the shape: halve the bytes per token, or double the bandwidth, and the ceiling doubles. Our inference from this: TFLOPS figures predict generation speed poorly, which is why unified memory systems are usually compared on bandwidth. Hugging Face makes the same point about the KV cache: during autoregressive decoding the required memory bandwidth for reloading keys and values "can become a serious time bottleneck".

What moves the number

Factor Effect Basis
Quantization Fewer bytes per weight, so less to read per token llama.cpp describes quantization as for "faster inference and reduced memory use"; the ceiling above follows by arithmetic
Batch size / concurrency Raises total system TPS until the GPU saturates; per-user speed does not rise with it NVIDIA benchmarking docs
Prompt length Raises TTFT NVIDIA benchmarking docs
Multi-query / grouped-query attention Less key and value data read from memory NVIDIA inference optimization blog
Speculative decoding A small draft model proposes tokens the large model verifies; the original paper reports 2X-3X acceleration on T5-XXL with identical outputs Leviathan et al.
Model not fitting in VRAM Layers running from system memory slow decode Follows from the memory-bound argument; llama.cpp and Ollama support partial GPU offload

The speculative decoding figure comes from one model in one paper; it is not a general multiplier.

How to read a tok/s claim

  • Ask which phase. A prompt-processing number and a generation number are not comparable.
  • Ask for how many users. Aggregate server throughput and single-user speed answer different questions.
  • Check the model, precision and context length. The same hardware gives different results for a 16-bit and a 4-bit version of a model, and a long context adds KV cache reads to every step.
  • Compare like with like. Numbers from different engines, prompts and settings rarely transfer; run the same benchmark (for example, llama-bench with the same pp and tg settings) on both systems.

In the index

Sources

END OF ANALYSIS

Related Intelligence