Lokale LLMs mit Ollama ausführen: Kompletter Leitfaden

Why Run LLMs Locally?

• Privacy: Your data never leaves your machine.
• Cost: Zero per-token costs after setup.
• Speed: No network latency for inference.
• Offline: Works without internet connection.

Installing Ollama

• Download from [ollama.com](https://ollama.com) — available for macOS, Linux, and Windows.
• Run `ollama pull llama3` to download a model.
• Run `ollama run llama3` to start chatting.

Popular Models

• Llama 3 8B: Good general-purpose model, runs on 8GB RAM.
• Llama 3 70B: Much smarter, needs 48GB+ RAM.
• Qwen 2.5: Excellent for Chinese and multilingual tasks.
• Mistral 7B: Fast and efficient for coding tasks.
• Phi-3 Mini: Microsoft's small but capable model.

Using the API

Ollama provides a local REST API at `http://localhost:11434`.

• `POST /api/generate` — text generation.
• `POST /api/chat` — chat completions.
• Compatible with OpenAI SDK via base URL override.

Integration Tips

• Use with LangChain via `OllamaLLM` class.
• Use with Open WebUI for a ChatGPT-like interface.
• GPU acceleration: Works with NVIDIA (CUDA) and Apple Silicon (Metal).

Performance Expectations

• 7B model on M2 MacBook: ~30 tokens/second.
• 7B model on RTX 4090: ~60 tokens/second.
• 70B model on 48GB RAM (CPU): ~3 tokens/second.