Why Run LLMs Locally?
- Privacy: Your data never leaves your machine.
- Cost: Zero per-token costs after setup.
- Speed: No network latency for inference.
- Offline: Works without internet connection.
Installing Ollama
- Download from ollama.com — available for macOS, Linux, and Windows.
- Run `ollama pull gemma3` to download a model.
- Run `ollama run gemma3` to start chatting.
Popular Models
- Qwen3 8B: Well-rounded default, 5.2GB download, 40K context.
- Gemma 3 12B: Google's multimodal model, 8.2GB, 128K context.
- DeepSeek-R1 8B: Reasoning-focused distill, 5.2GB, 128K context.
- Qwen3-Coder 30B: Mixture-of-experts coding model, 19GB, wants a 24GB GPU.
- Phi-4 14B: Microsoft's compact reasoning model, 9.1GB, 16K context.
Sizes and context windows are the ones Ollama publishes for each tag (checked 2026-11-10).
Using the API
Ollama provides a local REST API at `http://localhost:11434`.
- `POST /api/generate` — text generation.
- `POST /api/chat` — chat completions.
- Compatible with OpenAI SDK via base URL override.
Both endpoints require a `model` field naming a tag you have already pulled (`ollama list` shows them), and the API reference marks it required. A minimal chat body is `{"model": "gemma3", "messages": [{"role": "user", "content": "hello"}]}`; send it to `/api/chat` and Ollama streams JSON objects back.
Add "stream": false to get a single object with the full reply and timing statistics.
Integration Tips
- Use with LangChain via `OllamaLLM` class.
- Use with Open WebUI for a ChatGPT-like interface.
- GPU acceleration: Works with NVIDIA (CUDA) and Apple Silicon (Metal).
Performance Expectations
- 7B model on M2 MacBook: ~30 tokens/second.
- 7B model on RTX 4090: ~60 tokens/second.
- 70B model on 48GB RAM (CPU): ~3 tokens/second. The 43GB download leaves little room for a long context.
Choosing the Right Local Model
Your choice depends on hardware and use case. For coding on a laptop, Qwen2.5-Coder 7B (4.7GB) or Llama 3.1 8B (4.9GB) at 4-bit quantization work great. For general conversation, Qwen3 8B (5.2GB) or Gemma 3 12B (8.2GB) strike a good balance. A 24GB GPU fits Qwen3-30B-A3B at 19GB; Llama 3.1 70B is a 43GB download, so it needs 48GB of memory or more, not 24GB.
Performance Tuning
- Quantization: Q4_K_M offers the best quality-to-size ratio. Q5_K_M is slightly better but 20% larger. Avoid Q2/Q3 unless absolutely necessary—quality drops sharply.
- Context length: Default is usually 2048 or 4096 tokens. Increase only if needed—longer contexts use more RAM.
- GPU offloading: Use --n-gpu-layers to offload as many layers as VRAM allows. More on GPU = faster.
- Batch size: Increase --batch-size for faster prompt processing if VRAM allows.
Common Pitfalls
The #1 mistake is running models too large for your hardware. A 70B model on 16GB RAM will be painfully slow (1–2 tokens/sec). Better to run a 7B model at 30+ tokens/sec. The #2 mistake is ignoring the system prompt—local models are very sensitive to it. A well-crafted system prompt can turn a mediocre model into a useful assistant.
Measure Tokens per Second Before You Choose a Model
The published parameter count tells you almost nothing about whether a model is usable on your machine. What matters is generation speed in tokens per second on your hardware, and that number depends on memory bandwidth far more than raw compute. A model that fits in memory but barely generates faster than you can read is frustrating; one that runs at thirty tokens per second feels like a conversation. Time it yourself with a fixed prompt of known output length before you commit to a size.
If a model does not fit in memory it does not run slower, it runs on the CPU, which is typically an order of magnitude worse. So the fit question is binary and comes first: check the memory requirement at your chosen quantisation against available memory, and leave room for the operating system and the context.
Quantisation Is the Main Lever
Dropping from sixteen-bit to eight-bit or four-bit weights roughly halves and quarters the memory footprint, and for many tasks the quality difference is small enough to be invisible in your own evaluation. It is not free: reasoning-heavy tasks and long-context retrieval degrade more than fluent conversation does. Test at the quantisation you intend to ship rather than assuming the top-of-file variant is what your users will see.
Where Local Models Genuinely Win
Privacy, offline operation, predictable cost, and freedom from rate limits. A local model that runs a classification task at five tokens per second and never leaves your laptop is better than a hosted one for work you cannot send anywhere. Be honest about the inverse too: for open-ended reasoning and code generation, the hosted frontier is still meaningfully ahead, and pretending otherwise costs your users quality.