Deploying Open-Source LLMs Locally: Ollama in Practice
Running large models on your own machine protects data privacy and makes offline development and experimentation easy. Ollama makes all of this as simple as installing an app.
ByAI Resource Hub
Why Choose Local Deployment
Local deployment means data never leaves your machine, which is ideal for handling sensitive material. It also has no API fees, making it easy to experiment repeatedly and integrate into local workflows.
Install and Run Your First Model
After installing Ollama from the official site, a single command pulls and runs a model. The first run automatically downloads the weights.
# Pull and chat directly
ollama run llama3.2
# Or just download the model
ollama pull qwen2.5Access via the API
Once started, Ollama exposes an HTTP interface on local port 11434, which you can integrate into your app just like calling a cloud API.
curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"messages": [{ "role": "user", "content": "Hi, please introduce yourself" }],
"stream": false
}'How to Choose the Right Model
Larger models generally perform better but demand more VRAM and memory. First pick a parameter size that suits your machine (such as 7B/8B), then balance speed against quality.
You can find many open-source models such as Llama, Qwen, Mistral, and Gemma in the Ollama model library and switch as needed.
Performance Tuning and GPU Tips
Ollama automatically uses GPU acceleration when available. For Apple Silicon Macs, Metal is used out of the box. On Linux with NVIDIA, make sure the CUDA drivers and the Ollama CUDA build are installed. If you run out of VRAM, try a smaller quantization (e.g., Q4_K_M instead of Q8_0) or a smaller model family. Setting OLLAMA_NUM_PARALLEL lets you handle concurrent requests on machines with enough memory.
Integrating with Your Application
Ollama’s API is compatible with the OpenAI chat completions format. This means you can reuse the same code by simply changing the base URL to http://localhost:11434/v1 and using any string as the API key. Libraries like LangChain and LlamaIndex also have native Ollama integrations, making it straightforward to swap a cloud model for a local one in your RAG or agent pipeline.