Build a Headless Ubuntu LLM Server with Ollama on 8 GB VRAM

By John published: August 18, 2026

Running large language models locally on your own hardware costs nothing per token, leaks no data to anyone’s API, and keeps working when your internet does not. This guide walks through building a permanent headless LLM inference server on Ubuntu Server 26.04 LTS, serving models with Ollama and a browser chat frontend with Open WebUI, on a single NVIDIA GPU with 8 GB of VRAM.

Everything here is aimed at a machine that lives on a shelf with no monitor attached and comes back on its own after a power cut. That constraint drives most of the decisions, from the Secure Boot settings to which packages get held back from automatic updates.

What you can actually do with a small local LLM

An 8B model on a single consumer GPU is not going to write your novel or replace a frontier model for hard reasoning. What it will do is sit on your network answering an unlimited number of requests for free, without a single word of it leaving the building. That combination — good enough, private, and metered by nothing — turns out to be useful in ways a metered cloud API is not.

• Ask questions of your own documents. Point it at insurance policies, appliance manuals, tenancy agreements, tax paperwork, medical letters — the pile of PDFs everyone has and nobody reads. Local means you can feed it the sensitive stuff you would never paste into a website.
• Summarise the unreadably long. Meeting transcripts, 90-page planning applications, YouTube auto-captions, a decade of your own notes. Length is cheap when you are not paying per token.
• A coding assistant you can leave running all day. Editor plugins that speak the OpenAI API point straight at this box: explain a function, write the boring test, refactor a loop. No subscription, no per-seat licence, no code leaving your machine — which matters if your employer’s contract says it cannot.
• Bulk text work nobody wants to do by hand. Extract fields from a folder of receipts into a spreadsheet, tag and rename thousands of notes, write alt text for a photo library, normalise messy log or CSV data. Tasks where you need language understanding a thousand times over and the per-call cost would otherwise stop you.
• Natural language control of your home. Home Assistant and similar tools can hand interpretation of a spoken command to a local model, so “turn the heating down and dim the living room” works without a cloud assistant listening to your house.
• Writing help that works offline. Drafting, tightening, translating, or rewriting something in a kinder tone — on a train, on a boat, or during a broadband outage.
• A sandbox for building things. Prototype a RAG pipeline or an agent loop and hammer it ten thousand times while you get it wrong. No bill, no rate limit, no anxiety about a runaway script.
• Household use without an account for everyone. Homework explanations and general questions for the family, with no vendor account, no age gate, and no data collection.

Where it falls short is worth saying plainly: an 8B model is weaker on obscure factual recall, long multi-step reasoning, and anything needing a large codebase in context. Treat it as a fast, private, tireless text tool rather than an oracle, and keep a frontier model bookmarked for the hard problems.

Everything above talks to the same OpenAI-compatible endpoint covered in Part 7, so most existing tools need nothing more than a changed base URL.

What you’re building

Throughout this guide the example host is called llm-host on a LAN domain of homelab.lan, at 192.168.1.20. Those are placeholders — substitute your own names and addressing. The DHCP and DNS steps are shown on OPNsense; adapt them to whatever your own router or firewall runs.

When the build is finished you have two endpoints on your LAN:

Part 0 — Design decisions, answered up front

Can a local LLM server run completely headless?

Yes, completely. Ubuntu Server ships no display server at all, so the finished machine is headless by definition. The only question is whether you need a monitor during the install, and there are two answers:

• One-time monitor and keyboard. Plug a screen in, run the installer, enable SSH with your public key, unplug the screen permanently. Simplest and least error-prone. Recommended unless the box is somewhere awkward.
• Fully unattended autoinstall. Ubuntu’s installer accepts a cloud-init user-data file, so you can boot the ISO and have it partition, install, set the hostname, and drop in your SSH key with nobody at the keyboard. Covered in Part 1.3b.

One BIOS caveat either way: some boards halt POST if they detect no keyboard, and a few refuse to boot with no display device. Check for a “Halt On” or “Wait for F1 on error” setting and turn it off while you still have a screen attached.

Do you need a second GPU to free up the main one?

No. This is the main reason to run Ubuntu Server rather than Ubuntu Desktop.

With no X or Wayland session running, the GPU holds almost nothing. The kernel keeps an EFI framebuffer for the text console, typically 5-20 MB, and the NVIDIA driver’s own overhead is a few tens of megabytes. Run nvidia-smi on a freshly booted headless server and you will see something in the region of 10-40 MiB used out of 8192 MiB. A GNOME desktop, by contrast, will take 300-600 MB before you have opened anything, and a browser with hardware acceleration can take a gigabyte on its own. Going headless recovers all of that without spending a penny.

If the CPU has an integrated GPU, set the iGPU as the primary display adaptor in BIOS. The console framebuffer then lives on the iGPU and the discrete card sits at genuinely zero. This is a nice-to-have worth about 20 MB, so do it if the option exists and forget about it if it does not.

The one legitimate reason to add a second card is to add VRAM. Ollama will split a model’s layers across multiple GPUs, so a second 8 GB card would let you run 14B-class models entirely in VRAM instead of spilling to CPU. That is a capability upgrade, not a resource-contention fix, and it comes with its own headaches: PCIe lane allocation, PSU headroom, and the fact that layer-split inference runs at roughly the speed of the slower card. Park it as a future option.

What 8 GB of VRAM and 32 GB of RAM actually buys you

The 8 GB tier lands squarely on the 7-9B parameter class at Q4_K_M quantisation. Budget it like this:

Model weights (8B @ Q4_K_M)      ~4.7 GB
KV cache (8K context, fp16)      ~1.0 GB
Compute / CUDA context buffers   ~0.6 GB
Console framebuffer + driver     ~0.04 GB
                                 --------
Total                            ~6.3 GB of 8 GB

Two things follow.

Context length is what kills you, not the weights. The KV cache scales linearly with context. A model that loads fine at 4K will fall out of VRAM at 32K, and Ollama will silently offload layers to CPU, dropping you from 40+ tokens per second to single digits. Cap context deliberately rather than discovering the cliff. Quantising the KV cache to q8_0 roughly halves that line item and lets you run noticeably longer contexts for a barely perceptible quality cost.

14B-class models do not fit. A 14B at Q4_K_M is around 8-9 GB of weights alone. Ollama will happily pull it, load it, offload half the layers to system RAM, and crawl. If you want a 14B, you need 12 GB of VRAM.

The 32 GB of system RAM gives you three things: room for Ollama to offload layers when you deliberately want to run something oversized slowly, enough page cache to keep the model file resident so reloads are instant, and headroom for Docker, Open WebUI, and an embedding model running alongside.

Choosing a model: the class matters more than the name

This space moves fast enough that any specific model recommendation has a shelf life measured in weeks. The durable advice is the class — a 7-9B instruct model at Q4_K_M, from a recent generation. Check the Ollama model library for what is current rather than trusting a list in an article.

As of mid-2026, the Qwen 3.x 8-9B line, Llama 3.x 8B, and Gemma 3 9B are all sensible starting points, with a Qwen Coder variant worth having alongside if you write code. Pull two or three and benchmark them on your own prompts. Published benchmark rankings correlate weakly with whether a model is good at the thing you personally want.

Part 1 — Install Ubuntu Server 26.04 LTS for headless inference

Ubuntu 26.04 LTS (“Resolute Raccoon”) released in April 2026, with the 26.04.1 point release in August. If you would rather have a year of other people’s bug reports behind you, 24.04 LTS is supported to 2029 and every instruction here applies unchanged.

Download the Ubuntu Server ISO, not Desktop, then write it to a USB stick:

# Linux, replace sdX with the actual device. Check twice.
sudo dd if=ubuntu-26.04.1-live-server-amd64.iso of=/dev/sdX bs=4M status=progress oflag=sync

On Windows, use Rufus in DD mode. Balena Etcher works on any platform.

BIOS settings to change before you start

Secure Boot, MOK enrolment, and headless servers

The NVIDIA driver is an out-of-tree kernel module and must be signed to load under Secure Boot. Ubuntu handles this with MOK (Machine Owner Key) enrolment: the driver install prompts you to set a one-time password, and on the next boot a blue text screen appears demanding that password at the physical console. There is no way to answer it over SSH.

For a permanently headless box this is a landmine. Either disable Secure Boot in BIOS, or complete the MOK enrolment during your one-time physical session and never think about it again. Note that a future driver update can re-trigger enrolment if the key changes, which is a genuinely annoying way to discover your server did not come back from a reboot.

Check the current state at any time:

mokutil --sb-state

Guided install with a one-time monitor

Boot the USB, pick “Try or Install Ubuntu Server”, then work through:

• Network: leave on DHCP. You will pin the address with a reservation on OPNsense in Part 6, which keeps addressing in one place rather than split between the firewall and the host.
• Storage: see Part 2 for the layout. Use “Custom storage layout” rather than the default, which will hand your entire 2 TB to a single root filesystem and, worse, only actually allocate 100 GB of it to the LVM logical volume.
• Profile: your name, server name llm-host, username, password.
• SSH: tick Install OpenSSH server. Import your public key from GitHub or paste it. Do this now; it is what makes the machine headless.
• Featured snaps: select nothing. Docker gets installed properly from its own repository later, not as a snap.

After first boot, verify you can get in over the network before you unplug the monitor:

ssh youruser@<ip-address>

Autoinstall: a fully unattended Ubuntu Server build

If you want no physical session at all: write the ISO to one USB stick, then format a second stick with the volume label CIDATA (FAT32) containing two files.

meta-data (empty file, but it must exist):

touch meta-data

user-data:

#cloud-config
autoinstall:
  version: 1
  locale: en_GB.UTF-8
  keyboard:
    layout: gb
  identity:
    hostname: llm-host
    username: captain
    # Generate with: mkpasswd --method=SHA-512 --rounds=4096
    password: "$6$rounds=4096$REPLACE_THIS_WITH_A_REAL_HASH"
  ssh:
    install-server: true
    allow-pw: false
    authorized-keys:
      - ssh-ed25519 AAAAC3Nz... your-key-here
  storage:
    layout:
      name: lvm
      sizing-policy: all
  packages:
    - htop
    - curl
    - git
    - nvtop
  late-commands:
    - curtin in-target --target=/target -- systemctl enable ssh
  shutdown: reboot

Boot the installer USB and, at the GRUB menu, press e and append autoinstall to the linux line. The installer finds CIDATA, reads the config, and proceeds without prompting. sizing-policy: all is the important bit; it stops the default behaviour of leaving most of your disk unallocated.

Autoinstall is unforgiving of YAML errors and you get no feedback if it fails, so test the config in a VM first if the machine is genuinely inaccessible.

Part 2 — Storage layout for model files

You have a couple of terabytes. Split it deliberately.

If you have both an SSD and a spinning disk:

If you have one large SSD or NVMe: use LVM with a 150 GB root LV and a separate models LV for the rest. This lets you resize later without repartitioning.

Model files are read sequentially and only at load time, so a spinning disk is workable. A 5 GB model at roughly 150 MB/s takes about 35 seconds to load, and with a long keep-alive that happens once a day rather than once a request. NVMe brings it to 5 seconds. Put models on the fastest device you have room on, but do not buy hardware for it.

Create and mount the models volume:

sudo mkdir -p /srv/models
# Adjust the device path to match your setup
sudo mkfs.ext4 -L models /dev/sdb1
echo 'LABEL=models /srv/models ext4 defaults,noatime 0 2' | sudo tee -a /etc/fstab
sudo mount -a

noatime is worth having everywhere. It stops the kernel writing an access timestamp every time a file is read, which on a multi-gigabyte model file being memory-mapped is pure waste.

Add a modest swap file for safety even with 32 GB of RAM. It costs nothing and prevents the OOM killer from taking out your Ollama process during an ambitious model load:

sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Discourage the kernel from using it unless genuinely necessary
echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf

Part 3 — Base system configuration

Everything from here runs over SSH.

Update the system and set the hostname

sudo apt update && sudo apt full-upgrade -y

sudo hostnamectl set-hostname llm-host

Edit /etc/hosts so the fully qualified name resolves locally regardless of what DNS is doing:

sudo nano /etc/hosts

127.0.0.1       localhost
127.0.1.1       llm-host.homelab.lan llm-host
::1             localhost ip6-localhost ip6-loopback

Unattended security updates, and what to hold back

sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

Then hold the NVIDIA driver back from automatic updates. An unattended driver upgrade on a headless box with Secure Boot enabled is exactly the scenario that leaves you driving to the machine with a monitor under your arm:

sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

Add inside the package blacklist block:

Unattended-Upgrade::Package-Blacklist {
    "nvidia-";
    "libnvidia-";
    "linux-image";
    "linux-headers";
};

Blacklisting kernel packages too means you choose when to reboot into a new kernel and rebuild the DKMS module, rather than finding out at 3 a.m.

SSH hardening

sudo nano /etc/ssh/sshd_config

PasswordAuthentication no
PermitRootLogin no

sudo systemctl restart ssh

Confirm you can still log in from a second terminal before closing the first one.

Tooling worth installing

sudo apt install -y htop nvtop tmux curl jq git ncdu net-tools

nvtop is the one that matters. It gives you a live view of VRAM usage and GPU utilisation, which is how you will confirm models are actually running on the card.

Part 4 — Install the NVIDIA driver and verify GPU access

# See what Ubuntu recommends for your specific card
ubuntu-drivers devices

# Install it
sudo ubuntu-drivers install

sudo reboot

If Secure Boot is enabled, the install will prompt for a MOK password. Write it down, and be at the physical console on the next boot to enter it.

After reboot, verify:

nvidia-smi

You want to see your card, the driver version, a CUDA version, and memory usage in the tens of megabytes. If nvidia-smi reports “No devices were found” or “couldn’t communicate with the NVIDIA driver”, the module has not loaded. Check:

lsmod | grep nvidia          # should list nvidia, nvidia_uvm, nvidia_modeset
dmesg | grep -i nvidia       # look for signature verification failures
mokutil --sb-state           # Secure Boot blocking an unsigned module is the usual cause

Enable persistence mode so the driver stays initialised between requests. Without it, the driver tears down and re-initialises the GPU context each time the last process exits, adding a second or two of latency to the first request after an idle period:

sudo systemctl enable --now nvidia-persistenced

Part 5 — Install and configure Ollama

Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

The installer creates an ollama system user, installs to /usr/local/bin, and sets up a systemd service. It also detects the NVIDIA driver and pulls the right CUDA libraries.

Point Ollama at your models volume

sudo mkdir -p /srv/models
sudo chown -R ollama:ollama /srv/models
sudo chmod 755 /srv/models

Configure the Ollama systemd service

sudo systemctl edit ollama

Add this in the editable region:

[Service]
# Listen on all interfaces so the LAN can reach it
Environment="OLLAMA_HOST=0.0.0.0:11434"

# Model storage on the big volume
Environment="OLLAMA_MODELS=/srv/models"

# Keep the model resident for an hour after last use.
# Avoids reloading from disk on every conversation.
Environment="OLLAMA_KEEP_ALIVE=1h"

# With 8 GB VRAM, hold exactly one model at a time.
Environment="OLLAMA_MAX_LOADED_MODELS=1"

# Each parallel slot gets its own slice of the KV cache.
# On 8 GB, more than 2 will push you out of VRAM.
Environment="OLLAMA_NUM_PARALLEL=2"

# Flash attention reduces KV cache memory and speeds up long contexts
Environment="OLLAMA_FLASH_ATTENTION=1"

# Quantise the KV cache. Roughly halves its VRAM cost.
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

# Default context. Raise per-model once you have measured headroom.
Environment="OLLAMA_CONTEXT_LENGTH=8192"

# Allow browser-based clients on the LAN to call the API directly
Environment="OLLAMA_ORIGINS=*"

Apply:

sudo systemctl daemon-reload
sudo systemctl restart ollama
sudo systemctl status ollama

Confirm it is listening on all interfaces rather than just loopback:

ss -tlnp | grep 11434
# Expect 0.0.0.0:11434, not 127.0.0.1:11434

Pull a model and verify full GPU offload

ollama pull qwen3:8b        # substitute whatever is current
ollama run qwen3:8b "Say hello in one sentence."

Now the check that actually matters:

ollama ps

The PROCESSOR column must read 100% GPU. If it says anything with a CPU percentage in it, part of the model is running in system RAM and you are leaving most of your performance on the table. Fix it by dropping to a smaller model, reducing OLLAMA_CONTEXT_LENGTH, or checking that nothing else is holding VRAM with nvidia-smi.

Watch it live in a second SSH session:

nvtop

Tuning context length per model

The global OLLAMA_CONTEXT_LENGTH is a default. To raise it for one model, create a variant:

cat > /tmp/Modelfile <<'EOF'
FROM qwen3:8b
PARAMETER num_ctx 16384
EOF

ollama create qwen3-16k -f /tmp/Modelfile
ollama run qwen3-16k "test"
ollama ps    # confirm still 100% GPU

If it drops off 100% GPU, you have found your ceiling. Back off.

Part 6 — Give the server a name on your LAN

Two pieces: a fixed address, and a DNS record that points at it.

Fix the IP address with a DHCP reservation (OPNsense)

Get the MAC address of the machine’s active interface:

ip -brief link show

Then in the OPNsense web UI. Menu paths differ slightly between the ISC DHCP and Kea backends, so navigate by feature:

• Kea backend: Services → Kea DHCP → Kea DHCPv4 → Reservations → add a reservation with the MAC, your chosen IP, and hostname llm-host.
• ISC backend (legacy): Services → DHCPv4 → [LAN] → scroll to DHCP Static Mappings, add the MAC, IP, and hostname.

Choose an address outside the dynamic pool. If your pool is 192.168.1.100-199, use something like 192.168.1.20.

Reboot the server or release and renew its lease, then confirm:

ip -brief addr show

Add the DNS host override in Unbound

First check that System → Settings → General → Domain is set to homelab.lan. Then go to Services → Unbound DNS → Overrides → Host Overrides → Add:

Apply, then test from another machine on the LAN:

nslookup llm-host.homelab.lan
ping llm-host.homelab.lan

An explicit host override is more reliable than depending on the DHCP backend to register leases into Unbound automatically, which has been inconsistent across OPNsense versions and DHCP backends. Set it manually and it stays set.

Verify resolution on the server itself

If resolution behaves oddly on the Ubuntu box itself, check what systemd-resolved thinks:

resolvectl status
resolvectl query llm-host.homelab.lan

Part 7 — Using the OpenAI-compatible API endpoint

Ollama exposes an OpenAI-compatible API at /v1. This is the single most useful thing about the setup, because it means anything built against the OpenAI SDK points at your own box with a two-line change.

Available routes

Test the endpoint with curl

From any machine on the LAN:

curl http://llm-host.homelab.lan:11434/v1/models | jq

curl http://llm-host.homelab.lan:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:8b",
    "messages": [
      {"role": "system", "content": "You are terse and technical."},
      {"role": "user", "content": "Explain KV cache quantisation in three sentences."}
    ],
    "temperature": 0.7
  }' | jq -r '.choices[0].message.content'

Streaming:

curl -N http://llm-host.homelab.lan:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Count to ten."}],"stream":true}'

Calling your local LLM from Python

pip install openai

from openai import OpenAI

client = OpenAI(
    base_url="http://llm-host.homelab.lan:11434/v1",
    api_key="ollama",          # required by the SDK, ignored by Ollama
)

resp = client.chat.completions.create(
    model="qwen3:8b",
    messages=[
        {"role": "system", "content": "You are terse and technical."},
        {"role": "user", "content": "Explain KV cache quantisation."},
    ],
)
print(resp.choices[0].message.content)

Streaming:

stream = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Write a bash one-liner to find large files."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Calling it from Node

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://llm-host.homelab.lan:11434/v1",
  apiKey: "ollama",
});

const resp = await client.chat.completions.create({
  model: "qwen3:8b",
  messages: [{ role: "user", content: "Hello" }],
});

console.log(resp.choices[0].message.content);

Embeddings for local RAG

ollama pull nomic-embed-text

emb = client.embeddings.create(
    model="nomic-embed-text",
    input="Text to embed",
)
print(len(emb.data[0].embedding))   # 768

Embedding models are tiny, under 300 MB, and can sit in VRAM alongside a chat model without trouble.

Known differences from the real OpenAI API

Ollama’s compatibility layer is good but not total. Expect to hit these:

• logprobs is not supported.
• Function and tool calling depends on the model supporting it, and is less reliable than with frontier models. Test before building on it.
• The n parameter for multiple completions is ignored.
• Vision input works only with multimodal models, and the vision encoder adds around 1.4 GB of fixed VRAM overhead even for pure text prompts. On an 8 GB card, prefer a text-only build unless you actually need images.

Part 8 — Docker and Open WebUI

Install Docker from the official repository

Do not use the Ubuntu-packaged docker.io or the snap.

sudo apt install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker

If Ubuntu 26.04 is new enough that Docker has no matching codename in its repository, substitute the previous LTS codename (noble) in that echo line. It works fine.

The Open WebUI compose file

sudo mkdir -p /srv/docker/open-webui
cd /srv/docker/open-webui
sudo nano docker-compose.yml

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      # Bound to loopback only. nginx handles LAN exposure.
      - "127.0.0.1:8080:8080"
    environment:
      - OLLAMA_BASE_URL=http://host.docker.internal:11434
      - WEBUI_NAME=llm-host
      - WEBUI_URL=http://llm-host.homelab.lan:3000
      # First account created becomes admin. Set to false to
      # close registration once your accounts exist.
      - ENABLE_SIGNUP=true
      # Turn off outbound telemetry and update checks
      - SCARF_NO_ANALYTICS=true
      - DO_NOT_TRACK=true
      - ANONYMIZED_TELEMETRY=false
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui-data:/app/backend/data
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

volumes:
  open-webui-data:

docker compose up -d
docker compose logs -f

The host.docker.internal:host-gateway entry is what lets the container reach Ollama running on the host. Since Ollama is bound to 0.0.0.0, the container can reach it via the Docker bridge gateway address.

First start pulls a fairly large image and takes a few minutes. It is idle until you open it.

The log rotation block is worth keeping. Docker’s default json-file driver grows without limit, and an unattended container on a permanent install will eventually fill a partition with logs.

Part 9 — nginx reverse proxy

Basic auth is not strictly necessary for a LAN-only service, and skipping it is a defensible call. nginx still earns its place: it gives you a clean name-based entry point, handles websockets and long-running streams properly, sets sane upload limits, and gives you one place to add TLS later if you change your mind.

sudo apt install -y nginx
sudo nano /etc/nginx/sites-available/llm-host

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    listen 3000;
    server_name llm-host.homelab.lan llm-host;

    # Open WebUI accepts document uploads for RAG
    client_max_body_size 512M;

    access_log /var/log/nginx/llm-host.access.log;
    error_log  /var/log/nginx/llm-host.error.log;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;

        # Websockets - Open WebUI needs these for streaming
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Generation on a modest GPU can take a while.
        # The default 60s will cut long responses off mid-stream.
        proxy_read_timeout  600s;
        proxy_send_timeout  600s;

        # Stream tokens through immediately instead of buffering
        proxy_buffering off;
        proxy_cache off;
    }
}

Enable and reload:

sudo ln -s /etc/nginx/sites-available/llm-host /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl reload nginx

Open a browser on any LAN machine and go to http://llm-host.homelab.lan:3000.

Create the first account. It becomes the admin. Once your accounts exist, set ENABLE_SIGNUP=false in the compose file and run docker compose up -d to close registration, so a guest on your wifi cannot make themselves an account.

Port 80 is also listening, so plain http://llm-host.homelab.lan reaches the same place.

Optional: proxy the Ollama API through nginx

If you would rather clients hit a single port, add a second location block:

    location /ollama/ {
        proxy_pass http://127.0.0.1:11434/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }

The endpoint then becomes http://llm-host.homelab.lan:3000/ollama/v1. Direct access on 11434 is simpler and one less moving part, so this is a matter of taste.

Part 10 — Security posture for a LAN-only AI server

This build is LAN-only with no authentication on the frontend. That is defensible. Two things are worth stating explicitly so the decision stays an informed one.

Ollama’s API has no authentication whatsoever. Binding to 0.0.0.0 means any device on the LAN can not only run inference but also POST /api/pull to download arbitrary models onto your disk and DELETE /api/delete to remove them. On a trusted home LAN this is fine. On a LAN with guest devices, IoT tat, or anything you did not personally configure, it is a bit loose.

If you want to tighten it without adding complexity, bind Ollama to the LAN interface address rather than everything:

Environment="OLLAMA_HOST=192.168.1.20:11434"

Never port-forward 11434 or 3000 on your router. An open Ollama endpoint on the public internet gets found by scanners within hours and used as free compute. If you want remote access, use WireGuard, which OPNsense does natively and well. Set it up under VPN → WireGuard on the firewall, and you reach llm-host.homelab.lan from anywhere as though you were at home.

While you are in OPNsense, confirm there is no rule allowing WAN to LAN on those ports. The default deny should handle it, but confirming takes thirty seconds.

Part 11 — Verification checklist

Work through this after the build. Each line has a definite pass condition.

# 1. GPU visible and nearly empty
nvidia-smi
#    Expect: card listed, under 100 MiB used at idle

# 2. Ollama running and listening on the LAN
systemctl is-active ollama
ss -tlnp | grep 11434
#    Expect: active; 0.0.0.0:11434 or your LAN IP

# 3. Models on the right volume
ls -la /srv/models/blobs | head
df -h /srv/models

# 4. Model runs entirely on GPU
ollama ps
#    Expect: PROCESSOR column reads 100% GPU

# 5. API reachable from another machine
curl -s http://llm-host.homelab.lan:11434/v1/models | jq -r '.data[].id'

# 6. Name resolution works from another machine
nslookup llm-host.homelab.lan

# 7. Web UI reachable
curl -sI http://llm-host.homelab.lan:3000 | head -1
#    Expect: HTTP/1.1 200 OK

# 8. Container healthy
docker compose -f /srv/docker/open-webui/docker-compose.yml ps

# 9. Survives a reboot
sudo reboot
#    Then repeat 1-8 without touching anything

Item 9 is the one people skip and regret. A permanent headless install that does not come back cleanly from a power cut is not finished.

Part 12 — Operations and troubleshooting

Routine commands

# Live GPU view
nvtop

# What is loaded right now
ollama ps

# Everything on disk
ollama list

# Free VRAM immediately
ollama stop <model>

# Service logs
journalctl -u ollama -f
journalctl -u ollama --since "1 hour ago"

# Web UI logs
docker compose -f /srv/docker/open-webui/docker-compose.yml logs -f --tail 100

# Update the frontend
cd /srv/docker/open-webui
docker compose pull && docker compose up -d
docker image prune -f

Common failures and their fixes

Backing up what actually matters

The state worth keeping is small. Models are re-pullable, so skip them.

# Open WebUI database: conversations, users, settings
docker run --rm \
  -v open-webui_open-webui-data:/data \
  -v /srv/backups:/backup \
  alpine tar czf /backup/open-webui-$(date +%F).tar.gz -C /data .

# Config
sudo tar czf /srv/backups/config-$(date +%F).tar.gz \
  /etc/systemd/system/ollama.service.d/ \
  /etc/nginx/sites-available/llm-host \
  /srv/docker/open-webui/docker-compose.yml

Put that in a weekly cron job or systemd timer and push the result somewhere off the box.

Frequently asked questions

What can you actually do with a locally hosted LLM?

Private document question-answering, summarising long transcripts and PDFs, coding assistance in your editor, bulk text extraction and tagging, natural language home automation, and offline writing help. The practical advantage is not raw capability but the absence of metering and of data leaving your network, which makes high-volume and privacy-sensitive jobs viable.

Can you run an LLM server without a monitor or keyboard?

Yes. Ubuntu Server has no display server, so a finished install is headless by definition and managed entirely over SSH. You need a screen only during installation, and even that is avoidable with an autoinstall user-data file. The one thing to watch is Secure Boot: NVIDIA driver MOK enrolment demands a password at the physical console, which SSH cannot answer.

How much VRAM do you need to run an 8B model?

About 5.8 GB in practice — roughly 4.7 GB of weights at Q4_K_M, plus 0.5 GB of KV cache at 8K context with q8_0 quantisation, plus CUDA context overhead. That fits comfortably on an 8 GB card with room for a small embedding model alongside. Context length is the variable that pushes you over the edge, not the weights.

Can you run a 14B model on an 8 GB GPU?

Not usefully. A 14B at Q4_K_M is 8-9 GB of weights before any KV cache. Ollama will load it, offload half the layers to system RAM, and drop from 40+ tokens per second to single digits. 14B-class models want 12 GB of VRAM or more.

Is Ollama’s API really OpenAI-compatible?

Compatible enough that most code written for the OpenAI SDK works with a changed base_url and a dummy API key. The gaps are logprobs, the n parameter, and tool calling reliability, which depends on the model rather than on Ollama.

Does going headless actually free up VRAM?

Yes, and measurably. A headless Ubuntu Server install holds 10-40 MiB of VRAM for the console framebuffer and driver. A GNOME desktop session takes 300-600 MB before you open a single application. On an 8 GB card that difference is most of a quantisation step.

Is it safe to expose Ollama on a home LAN?

On a LAN you fully control, yes. Be aware that the Ollama API has no authentication at all, so any device that can reach port 11434 can pull and delete models as well as run inference. On a network with guest devices, bind Ollama to the LAN interface address rather than 0.0.0.0. Never port-forward it to the internet — use WireGuard for remote access instead.

Appendix A — Ollama environment variable reference

Appendix B — VRAM budgeting for an 8 GB GPU

Rough working figures at Q4_K_M quantisation:

Expect roughly 35-50 tokens per second for a fully-offloaded 8B model on a mid-range 8 GB card. Anything under 10 tokens per second means you are running on CPU, whatever the tool claims.

Appendix C — Ports in use

None of these should be reachable from the WAN side of the firewall.

Related guides

• Local AI guides — more on running models on your own hardware

This article may contain Amazon affiliate links. I may earn a small commission from qualifying purchases made through these links at no additional cost to you.