Quantization

Reducing the numeric precision of model weights (e.g. to 4-bit) to shrink size and run models on less hardware.

A 4-bit quantized model can be roughly a quarter the size of the 16-bit original, letting a 7B–8B model run on a laptop with a small, usually acceptable quality trade-off.

Related resources

Deploying Local LLMs with Ollama

A cross-platform tool that lets you download and run open-source LLMs like Llama, Mistral, and Phi locally on your own machine. It provides a CLI, a local API server, and an OpenAI-compatible endpoint, so you can experiment with models privately — no GPU or cloud account required. Ideal for developers who want offline AI capabilities or need to test prompts before deploying to production.

LM Studio Local Model Client

LM Studio is a desktop application that lets you download and run open-source LLMs locally with a single click. It provides a polished graphical interface for chatting with models, an OpenAI-compatible local API server, and support for GGUF/GGML formats — no command line or GPU expertise required.

llama.cpp is the lightweight C/C++ inference engine that runs open LLMs efficiently on CPUs and consumer GPUs, serving as the foundation beneath many local AI tools like Ollama and LM Studio. Its GGUF quantization formats let large models fit into modest hardware without dedicated accelerators.