What is an LLM inference (serving) engine?
An inference engine is the software that actually runs a model's forward pass and serves tokens — the layer between the model weights and your application. Choosing one is now a bigger practical decision than choosing the model, because it sets your throughput, your latency and how many users one GPU can serve.
The main options
| Engine | Built for | Notes |
|---|---|---|
| Ollama / llama.cpp | One machine, one or few users | Easiest setup; GGUF quantised weights; the default for individuals |
| vLLM | Production, many concurrent users | Introduced PagedAttention (pages the KV cache so almost none is wasted) and continuous batching |
| SGLang | Production, heavy shared prefixes | RadixAttention keeps the KV cache in a tree and reuses shared prefixes across requests — strong for multi-turn chat and RAG |
| TGI | Hugging Face ecosystem | Now in maintenance mode; Hugging Face points new work at vLLM / SGLang / llama.cpp |
The idea that separates them
A naive server runs one request at a time and throws away the KV cache after each. Production engines keep the GPU full — adding and retiring requests mid-batch (continuous batching) and reusing cached computation (paged / radix attention). That is where the large throughput gaps between "a script" and "a serving engine" come from.
Picking one
Individual use or prototyping: Ollama. Serving a feature to real traffic: vLLM is the safe default; SGLang if your workload has large shared prompts. Benchmark on your hardware and your prompt shape — published numbers rarely transfer. See the guide: vLLM vs SGLang vs Ollama.