Skip to content

Inference

The process of running a trained model to produce outputs, as opposed to training it.

Inference speed is usually measured in tokens per second, and latency matters for chat UX. Serving engines like vLLM and hardware like Groq optimize inference throughput and cost.

Related resources

vLLM High-Performance Inference Framework
vLLM is a high-throughput inference and serving engine for large language models, using PagedAttention to dramatically improve memory efficiency and throughput. It supports continuous batching, tensor parallelism, and OpenAI-compatible API serving, making it a top choice for deploying LLMs at scale.
Groq High-Speed Inference API
Groq delivers ultra-fast LLM inference through its custom LPU hardware, offering API access to popular open-source models at remarkably low latency. It is ideal for real-time applications like chatbots and code assistants where speed matters, with pricing based on token usage.