Skip to content

Quantization

Reducing the numeric precision of model weights (e.g. to 4-bit) to shrink size and run models on less hardware.

Related resources

Deploying Local LLMs with Ollama
Run large language models locally with Ollama, supporting open-source models such as Llama and Mistral.
LM Studio Local Model Client
A desktop app that downloads and runs open-source LLMs locally with one click, offering a graphical interface and an OpenAI-compatible API.