Skip to content

Local AI Stack: Ollama + Open WebUI + Hermes Agent

A fully self-hosted local AI setup running on a single workstation — local LLM inference with Ollama, a browser chat UI with Open WebUI, and an autonomous, tool-using agent with Hermes Agent reachable from anywhere over Telegram.

Hardware used

  • Host: Alienware Aurora R9 (Intel i9-9900K)
  • GPU: NVIDIA RTX 2080 SUPER (8 GB VRAM)
  • OS: Ubuntu 24.04 LTS
  • Models: qwen3:8b / qwen2.5:7b (Q4_K_M)

8 GB of VRAM is the binding constraint throughout. Notes on what fits and where it spills to system RAM are called out below.

Why I built this

Most AI assistants today are cloud services — your prompts, your data, and your context all leave your network. For a homelab built around self-hosting and keeping data on my own hardware, that was the one piece I'd quietly outsourced without thinking about it.

I wanted to find out whether a genuinely useful AI agent — one that can search the web, run commands, remember context across sessions, and reach me on my phone — could run entirely on hardware I already own, with nothing leaving the network. Not a toy demo, but something I'd actually use day to day.

This page is the answer, written as a build guide so anyone can reproduce it. It includes the parts that didn't work the first time — the context-window limit, the tool-calling bug that needed a proxy to fix, and the honest tradeoffs of running models on a single 8 GB GPU. The stack works, and everything here transfers to bigger hardware when the homelab grows.

Before you start

You should be able to follow this end to end on a fresh Ubuntu machine. Here's what you need in place first.

Prerequisites

  • A Linux host — this guide uses Ubuntu 24.04 LTS. Other distros work with minor command changes.
  • An NVIDIA GPU with working drivers (optional but strongly recommended). Verify with nvidia-smi — it should list your card. CPU-only inference works but is much slower.
  • Docker installed — used for Open WebUI. Install with sudo apt install docker.io and add yourself to the docker group: sudo usermod -aG docker $USER (log out/in afterward).
  • curl and git — sudo apt install curl git.
  • A Telegram account — for the Hermes mobile gateway (optional; you can use the CLI instead).
  • ~20 GB free disk space — for models, containers, and caches.

Order matters

Install in the order below — Ollama first (everything depends on it), then Open WebUI and Hermes as independent front-ends. Verify each piece works before moving to the next; it's far easier to debug one layer at a time than to untangle a broken full stack.

Overview

Component Role Access
Ollama Local model server (OpenAI-compatible API) http://localhost:11434
Open WebUI Browser chat front-end http://localhost:3000 (or your mapped port)
Hermes Agent Autonomous tool-using agent + messaging gateway Telegram / CLI
LiteLLM Proxy that fixes Ollama's streaming + tool-call bug http://localhost:11435

The three core pieces are independent. Ollama is the foundation; Open WebUI and Hermes are two different front-ends that both talk to it.

1. Install Ollama

Ollama is the local inference server. It exposes an OpenAI-compatible API that everything else connects to.

curl -fsSL https://ollama.com/install.sh | sh

Verify the service is running:

systemctl status ollama
ollama --version

Pull a model

For an 8 GB card, a 7–8B model at Q4 quantization is the practical ceiling.

ollama pull qwen3:8b      # ~5.2 GB, strong tool-calling
# or
ollama pull qwen2.5:7b    # ~4.7 GB, lighter, more context headroom

Confirm it loads and responds:

ollama run qwen3:8b "Say hello in one sentence."

Keep the model warm

For an always-on agent, set these so the model stays loaded between requests and inference is faster:

OLLAMA_FLASH_ATTENTION=1     # faster attention, lower memory
OLLAMA_KV_CACHE_TYPE=q8_0    # ~halves KV cache size, keeps more in VRAM
OLLAMA_KEEP_ALIVE=-1         # never unload the model

Set them via systemctl edit ollama under a [Service] block as Environment="KEY=value", then systemctl restart ollama.

2. Install Open WebUI

Open WebUI is a polished browser chat interface. The cleanest install is the official Docker image.

docker run -d \
  --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 in a browser, create the first account (it becomes the admin), and Open WebUI auto-detects the local Ollama instance.

Container reaching Ollama

--add-host=host.docker.internal:host-gateway lets the container reach Ollama running on the host. If Open WebUI can't see your models, check that Ollama is bound to 0.0.0.0 (set OLLAMA_HOST=0.0.0.0 in the service env), not just loopback.

3. Install Hermes Agent

Hermes is the autonomous agent layer — it adds tool use (web search, terminal, file ops, memory), persistent memory, cron jobs, and messaging-platform gateways on top of any OpenAI-compatible model.

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | sh

The installer sets up uv, Python 3.11, and Node.js under ~/.hermes/, then launches an interactive setup wizard.

PATH gap

The installer may not add ~/.local/bin to your PATH. If the hermes command isn't found afterward, add this to ~/.bashrc:

export PATH="$HOME/.local/bin:$PATH"

Setup wizard choices

Walk through hermes setup and select:

  • Provider: Ollama (http://127.0.0.1:11434/v1)
  • Model: your pulled model (e.g. qwen3:8b)
  • Terminal backend: Local
  • Messaging: Telegram (paste a bot token from @BotFather, then lock the allowlist to your own Telegram user ID from @userinfobot)
  • Search: SearXNG, pointed at your instance (e.g. http://localhost:8888)

Run as a background service

systemctl --user restart hermes-gateway
loginctl enable-linger $USER   # keep it running after logout / on boot

Gotchas & Fixes

These are the non-obvious issues encountered during setup. Documenting them because each one cost real time.

Context window floor (64K minimum)

Hermes refuses models reporting under 64K context. Most local models default to 32K, which triggers:

ValueError: Model qwen2.5:7b has a context window of 32,768 tokens,
which is below the minimum 64,000 required by Hermes Agent.

Two ways to resolve

Option A — override in config. Set context_length under the model: block (top-level) in ~/.hermes/config.yaml. Note Hermes caches the detected value in ~/.hermes/context_length_cache.yaml; delete that file after editing config so it re-detects.

Option B — accept native context. A 7–8B model at a forced 64K spills past 8 GB VRAM into system RAM anyway. Running at the model's native 32K is often the better fit for the hardware.

Ollama streaming + tool-call bug (the big one)

The most important fix. Ollama hangs or mangles tool calls when streaming is on and many tool definitions are sent — which Hermes does on every request. The symptom is tools firing with broken arguments:

Tool memory returned error: {"error": "Unknown action 'None'"}
Tool web_search returned error: SearXNG returned HTTP 400  (empty q=)

This is not a model-size problem — it reproduces across 7B, 8B, and 14B models identically. The fix is a LiteLLM proxy that forces stream: false between Hermes and Ollama.

pip install 'litellm[proxy]' --break-system-packages

Create ~/litellm-config.yaml:

model_list:
  - model_name: qwen3-local
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434
      stream: false

Run the proxy:

litellm --config ~/litellm-config.yaml --port 11435

Then point Hermes at the proxy instead of Ollama directly via hermes model → Custom endpoint → http://127.0.0.1:11435/v1, model qwen3-local.

Result

With the proxy in place, tool calls return clean structured arguments — memory writes to ~/.hermes/memories/USER.md, web search fires with a real query, and the agent behaves reliably.

Disable thinking mode for cleaner tool calls

Qwen3's reasoning traces can confuse the tool-call parser. Bake a no-think variant with a Modelfile:

cat > qwen3-nothink.Modelfile << 'EOF'
FROM qwen3:8b
PARAMETER num_ctx 32768
SYSTEM /no_think
EOF

ollama create qwen3-nothink -f qwen3-nothink.Modelfile

Then reference qwen3-nothink in the LiteLLM config.

Persona via SOUL.md

For stable facts the agent should always know (without relying on the memory tool firing), edit ~/.hermes/SOUL.md. It's injected into the system prompt every session:

You are NikLabs Agent, a personal AI assistant running locally on the homelab.
Be direct, technical, and concise. The user's name is Nikhil...

Why small models (and what they can do)

The single biggest choice in this build is model size, and it's dictated by the 8 GB of VRAM on the RTX 2080 SUPER. Here's the reasoning, because it's the question anyone reproducing this will face first.

Why not a big model? A model has to fit in VRAM to run fast. At Q4 quantization, roughly:

Model Size (Q4) Fits 8 GB? Tool-calling Notes
qwen2.5:7b ~4.7 GB Yes Fair Lightest, most context headroom
qwen3:8b ~5.2 GB Tight Good Best balance on 8 GB
qwen3:14b ~9.3 GB No (spills to RAM) Excellent Runs, but slow on 8 GB
qwen3:32b ~20 GB No Excellent Needs 24 GB VRAM

A 14B model technically runs on 8 GB, but Ollama has to offload layers it can't fit into system RAM, and inference slows from seconds to 30–90 seconds per reply — too slow for an interactive assistant. A 32B model won't load meaningfully at all. So on 8 GB, a 7–8B model isn't a compromise, it's the right choice: it stays entirely in VRAM, responds quickly, and leaves room for context.

What a small (7–8B) model can do well:

  • Hold a natural conversation and answer general questions
  • Follow a defined persona (via SOUL.md)
  • Fire a single tool reliably — web search, a terminal command, a file read
  • Summarise a web result or a document
  • Save and recall facts from memory (with the LiteLLM proxy in place)

Where small models struggle (and why bigger ones are "better" at agents):

  • Chaining several tools in one task (search → summarise → save)
  • Long autonomous multi-step plans without losing track
  • Precise multi-argument tool calls under a heavy tool load
  • Sustained reasoning across many turns

In short: a 7–8B model is a capable assistant you ask things, not a fully autonomous agent that executes long plans. That distinction is what more VRAM and a bigger model buys you.

If you outgrow 8 GB

Reliable agentic tool-calling at the 14B tier wants 12–16 GB VRAM minimum, and 24 GB for the 27–32B models that approach cloud quality. The whole software stack above transfers unchanged to bigger hardware — only the model underneath changes. Start small, prove the pipeline, upgrade the GPU later.

Example use case: a homelab assistant on your phone

Here's a concrete thing this setup does well, within a small model's comfort zone — a single-tool task you can trigger from Telegram anywhere.

Goal: ask the agent for a quick status check on the homelab and get the answer on your phone.

From Telegram:

Run: df -h / and tell me how much disk space is free

The agent fires the terminal tool, runs the command on the host, reads the output, and replies in plain language — e.g. "Your root filesystem is 62% used, 38 GB free." One tool, one clean result, well within a 7–8B model's reliable range.

Other single-tool tasks that work reliably:

  • "Search for today's top tech news and summarise the top three." → web search
  • "Show me the contents of my nginx config." → file read
  • "What's the status of the ollama service?" → terminal (systemctl status)
  • "Remember that my QNAP NAS is at 192.168.1.x." → memory write

Build up gradually

Start with single-tool requests like these and confirm they're reliable before asking for anything that chains multiple tools. It's the realistic way to get dependable results from a local small model — and it's genuinely useful: a private, phone-accessible assistant that can check on and act on your own infrastructure, with nothing leaving the network.

Verify the build

If you've followed along, run these checks to confirm each layer works. This is how you know you've succeeded.

  1. Ollama responds: 

    ollama run qwen3:8b "Reply with the single word: ok"
    You should get a short reply.

  2. Open WebUI loads: visit http://localhost:3000, log in, and confirm your model appears in the dropdown and answers a test message.

  3. LiteLLM proxy is up: 

    curl http://localhost:11435/v1/models
    Returns JSON listing qwen3-local.

  4. Hermes gateway is running: 

    systemctl --user status hermes-gateway   # should be "active (running)"
hermes doctor                            # should be mostly green

  5. The agent uses tools — the real test. From Telegram (or the CLI), send: 

    Use your memory tool to save that my name is <yourname>
    Then check that the memory file was actually written: 
    ls -la ~/.hermes/memories/    # USER.md should now exist
    Ask "what is my name?" in a new message — it should answer correctly from memory. If it does, the full stack works end to end.

You're done

A working local AI agent: private, self-hosted, reachable from your phone, with tool use and persistent memory — and not a single token left your network.

Quick reference

# Ollama
ollama list                              # installed models
ollama ps                                # what's loaded in VRAM

# LiteLLM proxy (must be running for Hermes tool calls)
litellm --config ~/litellm-config.yaml --port 11435

# Hermes
hermes model                             # switch model/provider
hermes config edit                       # edit config.yaml
hermes logs errors | tail -30            # recent errors
hermes prompt-size                       # system-prompt + tool-schema breakdown
systemctl --user restart hermes-gateway  # restart the gateway

# Memory check
ls -la ~/.hermes/memories/               # USER.md / MEMORY.md appear when memory writes