Two pipelines make up a RAG system: ingestion (building the knowledge base) and retrieval (using it at query time).
Ingestion — preparing the open book
- Collect the raw data — PDFs, docs, spreadsheets, a whole website, a company database, whatever the source is.
- Extract the useful parts.
- Split it into chunks — small pieces, not the whole document at once.
- Convert each chunk into an embedding, a vector of numbers, since models work with numbers, not raw text.
- Store those embeddings in a vector database.
Worth being precise here: a vector database isn't just a database that happens to hold vectors. A normal database like MongoDB does keyword search — search "heart attack symptoms" and you only get documents with those exact words. A vector database does semantic search — search "heart attack symptoms" and it'll also surface a document about "cardiac arrest," because the meanings are close even though no words match. That's the entire point of embedding things in the first place.
Retrieval — using the open book at query time
- The user's query gets embedded, using the same embedding model used during ingestion.
- That embedding is used to search the vector database for the closest matches, semantically, not by keyword. Whatever comes back is the "context."
- The original query and the retrieved context get combined into one prompt — that's the "augmentation."
- The model generates an answer from that augmented prompt.
Retrieval, then augmentation, then generation. That's the full name, in order.
What actually matters when you build one
Three choices decide most of how well a RAG system performs:
- Chunking strategy — fixed-size chunks are simple but can cut a sentence in half mid-thought. Hierarchical chunking (by section or paragraph) is more common in production. Semantic chunking (new chunk when the topic shifts) is slower but higher quality. LangChain, LlamaIndex, and Haystack all have chunking utilities built in.
- Embedding model — OpenAI's Text Embedding 3, Gemini's embedding models, or open-source options like Sentence Transformers.
- Vector store — ChromaDB, Pinecone, ElasticSearch, or FAISS. One caveat worth knowing: FAISS is a library for similarity search, not a managed database — no built-in persistence or infra, you're wiring that up yourself. The others are closer to full databases out of the box.
Last part: the different shapes a RAG system can actually take, and which one fits which problem.