Securing a self-hosted LLM server

Running a model locally with Ollama, llama.cpp or vLLM is a few commands. Exposing that server safely is the part the quickstarts skip. The failure mode is consistent: a port bound to 0.0.0.0, no authentication, reachable from the internet.

The exposure checklist

Risk	Fix
Bound to all interfaces	Bind to 127.0.0.1 and reach it over SSH or a private network; never 0.0.0.0 on an untrusted network
No auth	Put a reverse proxy (Caddy, nginx, Traefik) in front with a token or mTLS — most local runners have no auth of their own
Auto-update over an unverified channel	Ollama for Windows shipped CVE-2026-42248: its updater did not verify signatures, letting an adjacent-network attacker push a malicious build. Patch, and disable auto-update until you are on a fixed version
Model pulled from anywhere	Treat a weights file as untrusted input; pull from known publishers; a malformed metadata file has caused out-of-bounds reads and server crashes
Runs as root with full host access	Run in a container as a non-root user, with a read-only filesystem where possible and no host network

The principles

1. Nothing listens on a public interface without auth in front of it. The model server is not the auth layer.
2. The weights file is data from a stranger. Pin the source; verify hashes.
3. Isolate the runtime. A container or VM, so a crash or an RCE in the server does not own the host.
4. Patch on a schedule. These runners move fast and security fixes land quietly in point releases.

The rule

If someone finds your inference endpoint, the worst case should be wasted GPU time — not host access, not lateral movement, not a data leak. If it could be worse than that, add a proxy and a sandbox until it is not.

Sources: CERT.pl, SentinelOne and The Hacker News coverage of the 2026 Ollama CVEs.