Build a Knowledge-Base Q&A System with RAG
LLMs do not know your private, up-to-date material and are prone to hallucination. RAG (retrieval-augmented generation) has the model answer based on a reliable knowledge base by "retrieving first, then generating"—one of the most practical production approaches today.
PorAI Resource Hub
Why RAG Is Needed
Injecting knowledge by fine-tuning the model directly is expensive and slow to update. RAG keeps knowledge in an external database and retrieves relevant snippets on demand to feed the model—cheap and always up to date.
Pipeline Overview
A typical RAG system has two phases: an offline "indexing phase" that processes documents into retrievable vectors, and an online "query phase" that retrieves and generates an answer based on the user question.
Document Chunking and Embeddings
First split documents into appropriately sized chunks. Too large and retrieval is imprecise; too small and context is lost. A common approach is a few hundred tokens with some overlap.
Then use an embedding model to turn each chunk into a vector and store it in a vector database (such as pgvector, Milvus, or Qdrant).
Vector Retrieval and Top-K
When a user asks a question, convert the question into a vector as well, run a similarity search in the database, and retrieve the most relevant chunks (Top-K).
You can combine keyword search for "hybrid retrieval" and use reranking to further improve recall quality.
Assemble Context and Generate the Answer
Insert the retrieved chunks into the prompt and instruct the model to "answer only based on the provided material, and state clearly if the material does not contain the answer"—this significantly reduces hallucination.
Advanced optimizations include query rewriting, citing sources, controlling context length, and deduplicating and filtering retrieval results.