OpenClaw: Local LLMs — Full Privacy, Zero Cost

OpenClaw supports local models out of the box via Ollama and llama.cpp. No API keys, no data in cloud — model runs on your hardware. Let's break down everything: from install to performance tuning.

1. Ollama — Easiest Way

Installation

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Windows
winget install ollama
# or download .msi from ollama.ai

Run Service

ollama serve
# Starts REST API at http://localhost:11434

Pull Models

# Top models for coding/general use
ollama pull llama3.1:8b        # balanced speed/quality, 4.7 GB
ollama pull nemotron3:8b       # best for coding (NVIDIA), 4.7 GB
ollama pull qwen2.5:7b         # multilingual, Chinese/English, 4.4 GB
ollama pull codellama:7b       # code-specialized, 3.8 GB
ollama pull phi3:mini          # fast, light (3.8B), 2.3 GB
ollama pull mistral:7b         # classic, good base, 4.1 GB
ollama pull gemma2:9b          # Google, strong, 5.4 GB

# For weak hardware (CPU only, 8-16 GB RAM)
ollama pull llama3.2:3b        # 2.0 GB
ollama pull phi3:mini          # 2.3 GB
ollama pull qwen2.5:3b         # 1.9 GB
ollama pull tinyllama:1.1b     # 637 MB — minimum

Verify

ollama list                    # installed models
ollama run llama3.1:8b         # test in terminal
ollama ps                      # running models

2. Connect to OpenClaw

Via Wizard (First Run)

  1. openclaw → Configure → LLM Provider → Ollama
  2. Base URL: http://localhost:11434 (default)
  3. Pick model from list (ollama list)
  4. Done!

Via config.toml

[llm]
provider = "ollama"
model = "llama3.1:8b"
base_url = "http://localhost:11434"
temperature = 0.3
max_tokens = 4096

# Quick model aliases
[llm.aliases]
code = "nemotron3:8b"
chat = "llama3.1:8b"
fast = "phi3:mini"
multi = "qwen2.5:7b"

Hot-switch: /model code, /model fast, /model qwen2.5:7b.

3. llama.cpp — Max CPU Performance

When to Use llama.cpp Over Ollama

  • Need max speed on CPU (AVX2, AVX-512, AMX)
  • \li>Want full control over quantization/params\li>Need models not in Ollama (specialized fine-tunes)\li>Integrate in own scripts without REST overhead

Install llama.cpp

# Package managers
brew install llama.cpp          # macOS
apt install llama.cpp           # Ubuntu/Debian
pacman -S llama.cpp             # Arch

# Build from source (max performance)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUBLAS=1  # for NVIDIA GPU
# or
make LLAMA_METAL=1   # for Apple Silicon GPU
# or
make LLAMA_VULKAN=1  # for AMD/Intel GPU

.gguf Models — Where to Get

Quantization — What to Pick

QuantSize (7B)QualitySpeedRAM
Q4_K_M~3.8 GB⭐⭐⭐⭐⭐⭐⭐⭐⭐6 GB
Q5_K_M~4.6 GB⭐⭐⭐⭐⭐⭐⭐⭐8 GB
Q8_0~7.2 GB⭐⭐⭐⭐⭐⭐⭐10 GB
Q3_K_M~3.1 GB⭐⭐⭐⭐⭐⭐⭐⭐⭐5 GB
Q2_K~2.7 GB⭐⭐⭐⭐⭐⭐⭐⭐4 GB

Recommendation: Q4_K_M — gold standard. Q5_K_M if RAM allows. Q3_K_M for weak machines.

Connect to OpenClaw

[llm]
provider = "llamacpp"
model_path = "~/models/nemotron3-8b-Q4_K_M.gguf"
n_gpu_layers = -1        # -1 = all layers on GPU (Metal/CUDA/Vulkan)
n_ctx = 8192             # context (8k, 16k, 32k, 128k)
n_threads = 8            # CPU threads (0 = auto)
temperature = 0.3
max_tokens = 4096

# Optional: grammars, logits, sampling
grammar = "json"
# or
json_schema = "{...}"

4. Performance: CPU vs GPU vs Metal

PlatformBackendModelsTips
Apple Silicon (M1/M2/M3)Metal (LLAMA_METAL=1)AllUnified memory — large models fly. n_gpu_layers = -1
NVIDIA GPU (CUDA)CUDA (LLAMA_CUBLAS=1)AllVRAM limit. n_gpu_layers = -1 (all layers). Q4_K_M fits in 8GB VRAM
AMD/Intel GPUVulkan (LLAMA_VULKAN=1)AllROCm for AMD. Vulkan for Intel Arc.
CPU OnlyBLAS / AVX2 / AMXUp to 13B Q4_K_Mn_threads = physical cores. AMX on new Intel = 2x speedup

5. Caching & Optimization

  • KV Cache — auto-enabled, stores attention keys/values between tokens
  • Context Shift — when n_ctx exceeded, old context shifts (not recomputed)
  • Flash Attention — enable flash_attn = true for speedup (needs GPU support)
  • MMap — mmap = true (default) — lazy model load from disk, saves RAM

6. Hot Model Switching

In Chat (if aliases configured)

/model code      → nemotron3:8b (coding)
/model chat      → llama3.1:8b (chat)
/model fast      → phi3:mini (fast)
/model multi     → qwen2.5:7b (multilingual)
/model custom    → path to .gguf file

In Config (config.toml)

[llm.aliases]
code = "nemotron3:8b"
chat = "llama3.1:8b"
fast = "phi3:mini"
multi = "qwen2.5:7b"
reasoning = "o1-preview"  # via OpenRouter/OpenAI

7. Top Models by Task (2024-2025)

TaskModel (Ollama)Model (llama.cpp .gguf)Size
Coding (best quality)nemotron3:8bnemotron3-8b-Q4_K_M.gguf4.7 GB
Coding (fast)codellama:7b / phi3:minicodellama-7b-Q4_K_M.gguf3.8 / 2.3 GB
General chat / refactoringllama3.1:8bllama-3.1-8b-Q4_K_M.gguf4.7 GB
Multilingual (RU/CN/ES)qwen2.5:7bqwen2.5-7b-Q4_K_M.gguf4.4 GB
Weak hardware (CPU, 8GB RAM)phi3:mini / llama3.2:3bphi3-mini-Q4_K_M.gguf2.3 / 2.0 GB
Max quality (16GB+ RAM)llama3.1:70b / nemotron3:70bnemotron3-70b-Q4_K_M.gguf39 GB

8. FAQ & Common Issues

  • Ollama: "connection refused" → ollama serve not running. Add to autostart.
  • llama.cpp: "out of memory" → reduce n_ctx, use Q3_K_M, disable mmap
  • Slow generation → check n_gpu_layers (should be -1 for GPU), enable flash_attn
  • Model not found → verify exact name: ollama list or full .gguf path
  • Context truncated → increase n_ctx (8192, 16384, 32768, 131072)
  • Slow model switching → keep frequent models in RAM (mmap=true), use aliases

9. Security: local_only mode

[privacy]
local_only = true  # blocks ALL external calls except local LLM
telemetry = false

In this mode OpenClaw never hits internet — even for model updates. Perfect for air-gapped environments.

Local LLM Setup Checklist ✅

  1. [ ] Install Ollama / llama.cpp
  2. [ ] Pull 2-3 models for tasks (code, chat, fast)
  3. [ ] Configure OpenClaw for Ollama (base_url) or llama.cpp (model_path)
  4. [ ] Test: /model code "hello"
  5. [ ] Configure aliases in config.toml for hot-switching
  6. [ ] Enable local_only = true for max privacy
  7. [ ] Test in messenger: "Lobster, write a Python function"

Links: Ollama Library | llama.cpp GitHub | TheBloke GGUF Models | bartowski Models | Discord #local-llm