Securing a self-hosted LLM server
Running a model locally with Ollama, llama.cpp or vLLM is a few commands. Exposing that server safely is the part the quickstarts skip. The failure mode is consistent: a port bound to 0.0.0.0, no authentication, reachable from the internet.
The exposure checklist
| Risk | Fix |
|---|---|
| Bound to all interfaces | Bind to 127.0.0.1 and reach it over SSH or a private network; never 0.0.0.0 on an untrusted network |
| No auth | Put a reverse proxy (Caddy, nginx, Traefik) in front with a token or mTLS — most local runners have no auth of their own |
| Auto-update over an unverified channel | Ollama for Windows shipped CVE-2026-42248: its updater did not verify signatures, letting an adjacent-network attacker push a malicious build. Patch, and disable auto-update until you are on a fixed version |
| Model pulled from anywhere | Treat a weights file as untrusted input; pull from known publishers; a malformed metadata file has caused out-of-bounds reads and server crashes |
| Runs as root with full host access | Run in a container as a non-root user, with a read-only filesystem where possible and no host network |
The principles
- Nothing listens on a public interface without auth in front of it. The model server is not the auth layer.
- The weights file is data from a stranger. Pin the source; verify hashes.
- Isolate the runtime. A container or VM, so a crash or an RCE in the server does not own the host.
- Patch on a schedule. These runners move fast and security fixes land quietly in point releases.
The rule
If someone finds your inference endpoint, the worst case should be wasted GPU time — not host access, not lateral movement, not a data leak. If it could be worse than that, add a proxy and a sandbox until it is not.
Sources: CERT.pl, SentinelOne and The Hacker News coverage of the 2026 Ollama CVEs.
END OF ANALYSIS