Why Run LLMs Locally?

  • Privacy: Your data never leaves your machine.
  • Cost: Zero per-token costs after setup.
  • Speed: No network latency for inference.
  • Offline: Works without internet connection.

Installing Ollama

  • Download from [ollama.com](https://ollama.com) — available for macOS, Linux, and Windows.
  • Run `ollama pull llama3` to download a model.
  • Run `ollama run llama3` to start chatting.

Popular Models

  • Llama 3 8B: Good general-purpose model, runs on 8GB RAM.
  • Llama 3 70B: Much smarter, needs 48GB+ RAM.
  • Qwen 2.5: Excellent for Chinese and multilingual tasks.
  • Mistral 7B: Fast and efficient for coding tasks.
  • Phi-3 Mini: Microsoft's small but capable model.

Using the API

Ollama provides a local REST API at `http://localhost:11434`.

  • `POST /api/generate` — text generation.
  • `POST /api/chat` — chat completions.
  • Compatible with OpenAI SDK via base URL override.

Integration Tips

  • Use with LangChain via `OllamaLLM` class.
  • Use with Open WebUI for a ChatGPT-like interface.
  • GPU acceleration: Works with NVIDIA (CUDA) and Apple Silicon (Metal).

Performance Expectations

  • 7B model on M2 MacBook: ~30 tokens/second.
  • 7B model on RTX 4090: ~60 tokens/second.
  • 70B model on 48GB RAM (CPU): ~3 tokens/second.

Choosing the Right Local Model

Your choice depends on hardware and use case. For coding on a laptop, Qwen2.5-Coder 7B or Llama 3.1 8B at 4-bit quantization work great. For general conversation, Mistral 7B v0.3 or Gemma 2 9B strike a good balance. With 24GB+ VRAM, try Llama 3.1 70B at Q4 for near-cloud quality.

Performance Tuning

  • Quantization: Q4_K_M offers the best quality-to-size ratio. Q5_K_M is slightly better but 20% larger. Avoid Q2/Q3 unless absolutely necessary—quality drops sharply.
  • Context length: Default is usually 2048 or 4096 tokens. Increase only if needed—longer contexts use more RAM.
  • GPU offloading: Use --n-gpu-layers to offload as many layers as VRAM allows. More on GPU = faster.
  • Batch size: Increase --batch-size for faster prompt processing if VRAM allows.

Common Pitfalls

The #1 mistake is running models too large for your hardware. A 70B model on 16GB RAM will be painfully slow (1–2 tokens/sec). Better to run a 7B model at 30+ tokens/sec. The #2 mistake is ignoring the system prompt—local models are very sensitive to it. A well-crafted system prompt can turn a mediocre model into a useful assistant.