OpenClaw: Local LLMs — Full Privacy, Zero Cost
OpenClaw supports local models out of the box via Ollama and llama.cpp. No API keys, no data in cloud — model runs on your hardware. Let's break down everything: from install to performance tuning.
1. Ollama — Easiest Way
Installation
# macOS
brew install ollama
# Linux
curl -fsSL https://ollama.ai/install.sh | sh
# Windows
winget install ollama
# or download .msi from ollama.ai
Run Service
ollama serve
# Starts REST API at http://localhost:11434
Pull Models
# Top models for coding/general use
ollama pull llama3.1:8b # balanced speed/quality, 4.7 GB
ollama pull nemotron3:8b # best for coding (NVIDIA), 4.7 GB
ollama pull qwen2.5:7b # multilingual, Chinese/English, 4.4 GB
ollama pull codellama:7b # code-specialized, 3.8 GB
ollama pull phi3:mini # fast, light (3.8B), 2.3 GB
ollama pull mistral:7b # classic, good base, 4.1 GB
ollama pull gemma2:9b # Google, strong, 5.4 GB
# For weak hardware (CPU only, 8-16 GB RAM)
ollama pull llama3.2:3b # 2.0 GB
ollama pull phi3:mini # 2.3 GB
ollama pull qwen2.5:3b # 1.9 GB
ollama pull tinyllama:1.1b # 637 MB — minimum
Verify
ollama list # installed models
ollama run llama3.1:8b # test in terminal
ollama ps # running models
2. Connect to OpenClaw
Via Wizard (First Run)
openclaw → Configure → LLM Provider → Ollama
- Base URL:
http://localhost:11434 (default)
- Pick model from list (ollama list)
- Done!
Via config.toml
[llm]
provider = "ollama"
model = "llama3.1:8b"
base_url = "http://localhost:11434"
temperature = 0.3
max_tokens = 4096
# Quick model aliases
[llm.aliases]
code = "nemotron3:8b"
chat = "llama3.1:8b"
fast = "phi3:mini"
multi = "qwen2.5:7b"
openclaw → Configure → LLM Provider → Ollamahttp://localhost:11434 (default)[llm]
provider = "ollama"
model = "llama3.1:8b"
base_url = "http://localhost:11434"
temperature = 0.3
max_tokens = 4096
# Quick model aliases
[llm.aliases]
code = "nemotron3:8b"
chat = "llama3.1:8b"
fast = "phi3:mini"
multi = "qwen2.5:7b"
Hot-switch: /model code, /model fast, /model qwen2.5:7b.
3. llama.cpp — Max CPU Performance
When to Use llama.cpp Over Ollama
- Need max speed on CPU (AVX2, AVX-512, AMX) \li>Want full control over quantization/params\li>Need models not in Ollama (specialized fine-tunes)\li>Integrate in own scripts without REST overhead
Install llama.cpp
# Package managers
brew install llama.cpp # macOS
apt install llama.cpp # Ubuntu/Debian
pacman -S llama.cpp # Arch
# Build from source (max performance)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUBLAS=1 # for NVIDIA GPU
# or
make LLAMA_METAL=1 # for Apple Silicon GPU
# or
make LLAMA_VULKAN=1 # for AMD/Intel GPU
.gguf Models — Where to Get
- TheBloke on HuggingFace — thousands of quantized models
- bartowski — new models (Qwen2.5, Nemotron, Phi-3.5)
- QuantFactory — large models with various quantizations
Quantization — What to Pick
| Quant | Size (7B) | Quality | Speed | RAM |
|---|---|---|---|---|
| Q4_K_M | ~3.8 GB | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | 6 GB |
| Q5_K_M | ~4.6 GB | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | 8 GB |
| Q8_0 | ~7.2 GB | ⭐⭐⭐⭐⭐ | ⭐⭐ | 10 GB |
| Q3_K_M | ~3.1 GB | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | 5 GB |
| Q2_K | ~2.7 GB | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ | 4 GB |
Recommendation: Q4_K_M — gold standard. Q5_K_M if RAM allows. Q3_K_M for weak machines.
Connect to OpenClaw
[llm]
provider = "llamacpp"
model_path = "~/models/nemotron3-8b-Q4_K_M.gguf"
n_gpu_layers = -1 # -1 = all layers on GPU (Metal/CUDA/Vulkan)
n_ctx = 8192 # context (8k, 16k, 32k, 128k)
n_threads = 8 # CPU threads (0 = auto)
temperature = 0.3
max_tokens = 4096
# Optional: grammars, logits, sampling
grammar = "json"
# or
json_schema = "{...}"
4. Performance: CPU vs GPU vs Metal
| Platform | Backend | Models | Tips |
|---|---|---|---|
| Apple Silicon (M1/M2/M3) | Metal (LLAMA_METAL=1) | All | Unified memory — large models fly. n_gpu_layers = -1 |
| NVIDIA GPU (CUDA) | CUDA (LLAMA_CUBLAS=1) | All | VRAM limit. n_gpu_layers = -1 (all layers). Q4_K_M fits in 8GB VRAM |
| AMD/Intel GPU | Vulkan (LLAMA_VULKAN=1) | All | ROCm for AMD. Vulkan for Intel Arc. |
| CPU Only | BLAS / AVX2 / AMX | Up to 13B Q4_K_M | n_threads = physical cores. AMX on new Intel = 2x speedup |
5. Caching & Optimization
- KV Cache — auto-enabled, stores attention keys/values between tokens
- Context Shift — when n_ctx exceeded, old context shifts (not recomputed)
- Flash Attention — enable
flash_attn = truefor speedup (needs GPU support) - MMap —
mmap = true(default) — lazy model load from disk, saves RAM
6. Hot Model Switching
In Chat (if aliases configured)
/model code → nemotron3:8b (coding)
/model chat → llama3.1:8b (chat)
/model fast → phi3:mini (fast)
/model multi → qwen2.5:7b (multilingual)
/model custom → path to .gguf file
In Config (config.toml)
[llm.aliases]
code = "nemotron3:8b"
chat = "llama3.1:8b"
fast = "phi3:mini"
multi = "qwen2.5:7b"
reasoning = "o1-preview" # via OpenRouter/OpenAI
7. Top Models by Task (2024-2025)
| Task | Model (Ollama) | Model (llama.cpp .gguf) | Size |
|---|---|---|---|
| Coding (best quality) | nemotron3:8b | nemotron3-8b-Q4_K_M.gguf | 4.7 GB |
| Coding (fast) | codellama:7b / phi3:mini | codellama-7b-Q4_K_M.gguf | 3.8 / 2.3 GB |
| General chat / refactoring | llama3.1:8b | llama-3.1-8b-Q4_K_M.gguf | 4.7 GB |
| Multilingual (RU/CN/ES) | qwen2.5:7b | qwen2.5-7b-Q4_K_M.gguf | 4.4 GB |
| Weak hardware (CPU, 8GB RAM) | phi3:mini / llama3.2:3b | phi3-mini-Q4_K_M.gguf | 2.3 / 2.0 GB |
| Max quality (16GB+ RAM) | llama3.1:70b / nemotron3:70b | nemotron3-70b-Q4_K_M.gguf | 39 GB |
8. FAQ & Common Issues
- Ollama: "connection refused" →
ollama servenot running. Add to autostart. - llama.cpp: "out of memory" → reduce n_ctx, use Q3_K_M, disable mmap
- Slow generation → check n_gpu_layers (should be -1 for GPU), enable flash_attn
- Model not found → verify exact name:
ollama listor full .gguf path - Context truncated → increase n_ctx (8192, 16384, 32768, 131072)
- Slow model switching → keep frequent models in RAM (mmap=true), use aliases
9. Security: local_only mode
[privacy]
local_only = true # blocks ALL external calls except local LLM
telemetry = false
In this mode OpenClaw never hits internet — even for model updates. Perfect for air-gapped environments.
Local LLM Setup Checklist ✅
- [ ] Install Ollama / llama.cpp
- [ ] Pull 2-3 models for tasks (code, chat, fast)
- [ ] Configure OpenClaw for Ollama (base_url) or llama.cpp (model_path)
- [ ] Test:
/model code "hello" - [ ] Configure aliases in config.toml for hot-switching
- [ ] Enable local_only = true for max privacy
- [ ] Test in messenger: "Lobster, write a Python function"
Links: Ollama Library | llama.cpp GitHub | TheBloke GGUF Models | bartowski Models | Discord #local-llm
