Local AI Security

Build the mental model

'Local' describes where inference happens, not how safe the surrounding system is. A local LLM server still listens on a network socket, and that socket has three meaningfully different exposure levels.

Exposure Level	What it means
localhost (127.0.0.1)	Only processes on the same machine can reach it.
LAN (0.0.0.0, private network)	Any device on the same Wi-Fi or switch can reach it — convenient for a homelab, but now anyone on that network is a potential client.
Public internet (forward/tunnel)	Anyone anywhere can reach it. At that point it needs the same defenses as any other internet-facing service: a firewall rule limiting ports/sources, authentication checked on every request, TLS, and usually a reverse proxy handling all of that instead of relying on the model server's own often-minimal built-in protections.

Exposure is only one axis. The content flowing through the system is another security dimension entirely.

Prompts from untrusted sources (a public form, a scraped document, a RAG chunk written by someone else) can contain instructions aimed at manipulating the model — anything with tool-calling ability needs permission boundaries independent of what the prompt says
Sensitive files should never sit in a directory an agent can read without cause
Request logs often contain full prompts and responses verbatim, making logs themselves a sensitive-data surface, not just a debugging convenience
The model file itself is a supply-chain risk — downloading weights from an unverified source is functionally similar to running an unverified binary

None of This Is Automatic

None of this is automatic just because the model runs on your hardware.

text

EXPOSURE LEVELS: SECURITY REQUIREMENT INCREASES
-----------------------------------------------
EXPOSURE LEVELS: SECURITY REQUIREMENT INCREASES
---------------------------------------------------

  localhost only        LAN                public internet
  (127.0.0.1)            (0.0.0.0, private)  (forwarded/tunneled)

    [you]                [you] [device2]     [anyone, anywhere]
      |                    |      |                  |
      v                    v      v                  v
    server               server                    server
    no auth needed       auth recommended          auth REQUIRED
    no TLS needed        TLS recommended           TLS REQUIRED
                                                    firewall + reverse
                                                    proxy + rate limits

  security requirement:  low ---------------------------> high

Connect it to a real scenario

The --host flag on most local inference servers (llama.cpp's server, text-generation-webui, Ollama's OLLAMA_HOST) is the single setting most responsible for how exposed the server is, and it's easy to change without realizing the consequence.

Flag	Effect
--host 127.0.0.1	Binds the server's socket so only processes on the same machine can open a connection — nothing on the network, even the same Wi-Fi, can reach it.
--host 0.0.0.0	Binds to every network interface the machine has, so the server becomes reachable from any other device on the LAN, and if the router forwards that port, from the public internet too.

No Default Authentication

Neither llama.cpp's built-in server nor Ollama's default setup ships with authentication turned on, so a 0.0.0.0 bind with no reverse proxy in front of it means anyone who can reach the IP and port can send inference requests, read whatever system prompt is embedded server-side, and consume your GPU.

The firewall rule pattern shown narrows LAN exposure to a specific subnet as a partial mitigation, but a reverse proxy adding real authentication and TLS is the actual fix, not a firewall rule alone — firewalls control who can reach the port, not who is authorized once they do.

Local does not mean secure — exposure level (localhost, LAN, or public), authentication, and TLS are separate decisions you still have to make.
Most local inference servers have no authentication by default, so the --host binding you choose is often the only thing standing between the model and an open network.
Model files, request logs, and tool permissions are all part of the security surface, not just the network port.

Local Does Not Mean Secure

Running a model on your own hardware controls where computation happens — it does not automatically add authentication, encrypt network traffic, verify where the model file came from, or restrict what a connected agent can do. Every one of those is a decision you still have to make explicitly; none of them happen by default just because there's no cloud provider in the loop.

Before Running an Unknown Model

Source is a trusted, well-known repository (e.g. the model author's own Hugging Face org, not a random re-upload).
Repository name and file match what you actually intended to download — no typo-squatted or near-identical names.
Model card has been read, including any stated limitations, biases, or intended-use restrictions.
License has been reviewed and matches how you intend to use the model (personal, research, or commercial).
File format is confirmed (GGUF, safetensors, etc.) and matches what your runtime expects.
Quantization level is confirmed and matches the tradeoff you want between quality, speed, and memory.
Runtime/engine compatibility is confirmed (e.g. GGUF version supported by your llama.cpp build).
Hardware requirements (VRAM/RAM) are estimated against your actual machine before downloading.
Enough free disk space is confirmed for the full download plus any conversion or cache files.

Try the working example

bash

# Bind only to localhost -- reachable only from this machine
python -m llama_cpp.server --host 127.0.0.1 --port 8080 --model ./model.gguf

# Bind to all interfaces -- reachable from any device on the LAN
# (or the public internet, if the port is forwarded on your router)
python -m llama_cpp.server --host 0.0.0.0 --port 8080 --model ./model.gguf

# Safer LAN exposure: bind to the LAN interface explicitly, then put a
# reverse proxy in front for TLS + auth, and firewall the raw port
python -m llama_cpp.server --host 192.168.1.50 --port 8080 --model ./model.gguf
sudo ufw allow from 192.168.1.0/24 to any port 8080 proto tcp
sudo ufw deny 8080

You should see

Both commands start immediately and print a startup banner along with the address they're listening on — there's no numeric benchmark output to show. The practical difference is only observable from another machine: with the 127.0.0.1 bind, a curl request from any other device on the network fails to connect (connection refused), because the OS never accepts an incoming connection on that socket from outside the machine. With the 0.0.0.0 bind, the same curl request from another device on the LAN succeeds and reaches the model, because the socket is now listening on the machine's real network interface rather than only the loopback interface. The ufw rules narrow that further so only a specific subnet can reach the port at all, but they don't add authentication — a device inside the allowed subnet still gets unauthenticated access to the raw inference API.

5-minute try-it

On a machine you control, start a local inference server bound to 127.0.0.1 and try to reach it with curl from a second device on the same network — confirm the connection is refused. Then restart it bound to your machine's LAN IP (not 0.0.0.0, to keep it explicit) and repeat the curl from the second device. Note the difference, then add a firewall rule that only allows your own subnet before leaving it running.

One important caution

Binding to 0.0.0.0 'just to test from my phone' and forgetting to close it — most local model servers have no authentication on by default, so this quietly leaves the model open to the whole LAN.

Treating a firewall rule as equivalent to authentication — a firewall controls which network a request can come from, not whether the request itself is authorized once it's inside that network.

Hugging Face Hub — Security — Local AI / Local LLM