Build a Knowledge-Base Q&A System with RAG
LLMs do not know your private, up-to-date material and are prone to hallucination. RAG (retrieval-augmented generation) has the model answer based on a reliable knowledge base by "retrieving first, then generating"—one of the most practical production approaches today.
ByAI Resource Hub
Why RAG Is Needed
Injecting knowledge by fine-tuning the model directly is expensive and slow to update. RAG keeps knowledge in an external database and retrieves relevant snippets on demand to feed the model—cheap and always up to date.
Pipeline Overview
A typical RAG system has two phases: an offline "indexing phase" that processes documents into retrievable vectors, and an online "query phase" that retrieves and generates an answer based on the user question.
Document Chunking and Embeddings
First split documents into appropriately sized chunks. Too large and retrieval is imprecise; too small and context is lost. A common approach is a few hundred tokens with some overlap.
Then use an embedding model to turn each chunk into a vector and store it in a vector database (such as pgvector, Milvus, or Qdrant).
Vector Retrieval and Top-K
When a user asks a question, convert the question into a vector as well, run a similarity search in the database, and retrieve the most relevant chunks (Top-K).
You can combine keyword search for "hybrid retrieval" and use reranking to further improve recall quality.
Assemble Context and Generate the Answer
Insert the retrieved chunks into the prompt and instruct the model to "answer only based on the provided material, and state clearly if the material does not contain the answer"—this significantly reduces hallucination.
Advanced optimizations include query rewriting, citing sources, controlling context length, and deduplicating and filtering retrieval results.
Choosing a Vector Database
The vector database sits at the heart of RAG. For prototyping, Chroma runs in-memory with zero setup. For production workloads, Qdrant (Rust-based) and Pinecone (fully managed) are designed for production deployment; check current limits and scaling options in their documentation. If you already run PostgreSQL, pgvector avoids adding new infrastructure. Weaviate shines when you need built-in vectorization modules and a GraphQL API.
Evaluating RAG Quality
A RAG system is only as good as its retrieval and its generation. Measure retrieval with Recall@K (does the right chunk appear in the top-K results?) and measure generation with faithfulness (does the answer stay grounded in the retrieved context?) and relevance (does it actually answer the question?). Frameworks like RAGAS automate these metrics so you can track quality as you iterate on chunking strategies, embedding models, and prompt templates.
Before You Build: Prerequisites
You need a small, representative document set, permission to use it, and a parser that extracts clean text (PDFs and scanned files often need OCR). Choose an embedding model and a vector store you can run in your environment, and confirm their dimension and language support. Prepare 20–50 real questions with the document or passage that should answer each one. Finally, decide how documents are updated and who can read which collection, because permissions must be enforced at retrieval time, not only in the UI.
You do not need a large corpus to start. A few hundred well-structured pages are enough to validate the pipeline before scaling.
Verification: Check Retrieval Before You Check Answers
Test retrieval first. For each labeled question, run the search and inspect the top results: the expected passage should appear in the top-K, and unrelated chunks should not dominate. If it does not, adjust chunk size, overlap, metadata filters, or the embedding model before touching the generation prompt. Then check answers: the model should cite or quote the retrieved passage, and say the material does not contain the answer when it truly does not. Keep a fixed evaluation set and rerun it after every change so you can see regressions. Also test a question whose answer is absent, a question with two similar documents, and a question in each language you support.
Troubleshooting Common RAG Failures
Debug in this order. If retrieval returns nothing, check that documents were indexed, that query and document embeddings use the same model and dimensions, and that filters are not excluding the collection. If the right chunk exists but ranks low, inspect chunk boundaries and try hybrid keyword search or reranking. If the answer ignores the retrieved context, confirm the chunks actually reached the prompt and were not truncated; then tighten the instruction to answer only from the provided material and cite sources. If answers are stale, verify the reindex job ran after the document update. If duplicates fill the top-K, deduplicate by document ID or content hash before inserting. Finally, verify that access control filters run during retrieval, not after generation.
When RAG Is the Wrong Tool
RAG is a poor fit when the entire knowledge base fits comfortably in the model’s context window and changes rarely; passing the documents directly is simpler. It is also weak when the source material is unreliable, contradictory, or mostly images without usable text. If the task requires calculation, transactions, or live state, call a database or API instead of retrieving prose about it. For questions that need a single authoritative lookup, a search index or structured query may be more accurate. And if no one can maintain the index, a retrieval system will decay quickly; a smaller curated document may serve users better.
One chunking step
This function solves the indexing step that splits a document before you embed it. It is a starting sketch, not a measured result. Failure: if overlap is 0, a fact that sits on the cut is stored in neither chunk, and retrieval cannot return it.
def chunk_text(text, size=400, overlap=50):
chunks = []
start = 0
while start < len(text):
chunks.append(text[start:start + size])
if start + size >= len(text):
break
start += size - overlap
return chunks