Skip to content

GGUF

A file format for storing quantized models, widely used to run LLMs locally with tools like llama.cpp and Ollama.

A single .gguf file bundles the weights and metadata, and filenames usually mark the quantization level (e.g. Q4_K_M). Higher-bit files are more accurate but larger and slower.

Related resources

Deploying Local LLMs with Ollama
A cross-platform tool that lets you download and run open-source LLMs like Llama, Mistral, and Phi locally on your own machine. It provides a CLI, a local API server, and an OpenAI-compatible endpoint, so you can experiment with models privately — no GPU or cloud account required. Ideal for developers who want offline AI capabilities or need to test prompts before deploying to production.
LM Studio Local Model Client
LM Studio is a desktop application that lets you download and run open-source LLMs locally with a single click. It provides a polished graphical interface for chatting with models, an OpenAI-compatible local API server, and support for GGUF/GGML formats — no command line or GPU expertise required.
llama.cpp
llama.cpp is the lightweight C/C++ inference engine that runs open LLMs efficiently on CPUs and consumer GPUs, serving as the foundation beneath many local AI tools like Ollama and LM Studio. Its GGUF quantization formats let large models fit into modest hardware without dedicated accelerators.