Skip to content
Fathom Layer
GlossaryGlossary7 MIN READ

What is an LLM inference (serving) engine?

Compute server connected to a switch beside wireless clients with conceptual response panels
AI-generated editorial illustration · Conceptual scene

An LLM inference engine is the software that loads a language model's weights, runs the model to generate tokens, and serves the results to applications, usually over an HTTP API. It decides how GPU memory is used, how requests are scheduled and batched, and which hardware you can run on, so the same model can behave very differently under two engines. "LLM serving" usually refers to engines built for many concurrent users on a server; local engines optimize for one machine and one or a few users.

How it works

Every engine does the same basic job: tokenize the prompt, run the prefill pass over it, then decode one token at a time while storing attention keys and values in a KV cache. What separates engines is what they do around that loop.

  • Memory management for the KV cache. The vLLM paper found that in earlier serving systems only 20.4% to 38.2% of KV cache memory held actual token states, with the rest lost to fragmentation and over-reservation. Its answer, PagedAttention, borrows the idea of virtual memory paging from operating systems to allocate the cache in blocks, achieving what the authors call "near-zero waste".
  • Scheduling. Instead of waiting for a whole batch to finish, serving engines add and retire requests as they go. The vLLM paper cites iteration-level scheduling (Yu et al., 2022, the Orca system) as prior work; vLLM calls its version continuous batching, and NVIDIA's TensorRT-LLM calls it in-flight batching.
  • Reuse of shared prompts. When many requests share a prefix, such as the same system prompt, an engine can reuse the cached keys and values instead of recomputing them. SGLang's RadixAttention is built for this.

Because decoding is memory-bound (see tokens per second), we infer that these memory and scheduling choices matter more to throughput than raw compute; that is our reading, not a measured result.

The main engines, from their own documentation

Engine Description (quoted or paraphrased from its docs) Notable features (per official docs)
vLLM "A fast and easy-to-use library for LLM inference and serving" PagedAttention; continuous batching; OpenAI-compatible API server; NVIDIA and AMD GPUs, x86/ARM/PowerPC CPUs, and hardware plugins including Apple Silicon
SGLang "High-performance serving framework for large language and multimodal models" RadixAttention, prefix caching, multi-GPU parallelism
TensorRT-LLM NVIDIA's "open-source library for accelerating and optimizing inference" on NVIDIA GPUs In-flight batching, paged KV cache, speculative decoding, chunked prefill, quantization
Text Generation Inference (TGI) Hugging Face's serving toolkit, now in maintenance mode Hugging Face recommends vLLM, SGLang, and local engines such as llama.cpp or MLX going forward
llama.cpp "Plain C/C++ implementation without any dependencies" 1.5-bit to 8-bit quantization; Apple silicon "a first-class citizen"; CUDA, HIP, Vulkan and SYCL backends; CPU+GPU hybrid inference; server with OpenAI-compatible routes and parallel decoding
Ollama Runs models through a "Local Ollama server" (its API docs) Local API at localhost:11434; supports a subset of the OpenAI API; models stay in memory 5 minutes by default

llama.cpp uses the GGUF file format, which Hugging Face describes as a binary format designed for GGML-based executors that stores tensors together with standardized metadata.

Performance claims. The vLLM authors report 2-4x higher throughput than FasterTransformer and Orca at the same latency. The SGLang authors report up to 6.4x higher throughput than state-of-the-art systems on the tasks they tested. NVIDIA's TensorRT-LLM overview claims "World-record inference performance on Blackwell GPUs" for DeepSeek R1. Each claim comes from the team that built the engine, measured on its own chosen workloads, and each team is promoting the engine it maintains. This index has not benchmarked them.

Server engine or local engine

Serving engines (vLLM, SGLang, TensorRT-LLM) Local engines (llama.cpp, Ollama)
Target Many concurrent users on data-center GPUs One machine, one or a few users
Hardware GPU-first; TensorRT-LLM is NVIDIA-only CPUs, consumer GPUs, Apple silicon
Memory policy vLLM reserves a fraction of GPU memory up front (--gpu-memory-utilization, default 0.92) Ollama loads and unloads models on demand
Model larger than GPU memory Configure parallelism across GPUs per each engine's docs llama.cpp can split a model between CPU and GPU when it exceeds VRAM
Operations More configuration: parallelism, memory, batching Few settings; Ollama binds to 127.0.0.1 by default

The table describes design targets, not hard limits: vLLM also runs on CPUs, and llama.cpp's server supports multiple users.

When to use which

  • Personal use, prototyping, offline work: a local engine. Ollama exposes a local API on localhost:11434 once installed; llama.cpp exposes options for quantization, GPU offload and KV cache types.
  • A product feature with real concurrent traffic: a serving engine. vLLM documents support for NVIDIA and AMD GPUs and several CPU architectures; SGLang describes itself as "designed for low-latency, high-throughput inference with RadixAttention, prefix caching, and multi-GPU parallelism"; TensorRT-LLM targets teams committed to NVIDIA hardware.
  • New deployments on TGI: Hugging Face itself now points users to other engines.

When the choice matters less

Our inference, not a measurement: with a single user on one machine, batching and scheduling have little to work with, so the practical differences shrink to hardware support, quantization formats and convenience. Concurrency is where serving engines earn their complexity.

Honest caveats

  • Published throughput numbers depend on model, hardware, prompt length, output length and concurrency. Benchmark your own workload.
  • Engines change quickly. Defaults cited here (vLLM's 0.92 memory fraction, Ollama's 5-minute keep-alive) were read from the official docs on 29 September 2026 and can change between releases. Ollama's own pages disagree on default context: its FAQ says 4,096 tokens, while its context-length page says the default scales with VRAM (4k under 24 GiB, 32k at 24-48 GiB, 256k at 48 GiB or more).
  • Ollama's documentation notes that parallel requests multiply context: 4 parallel requests at a 2K context allocate an 8K context, which costs memory.

In the index

Sources

END OF ANALYSIS

In the Fathom index

Tools in the index for running language models locally. Matched on each item's own record and sorted A–Z, not by merit.

Browse Developer tools →

Related Intelligence