RAG: how to make an LLM answer with your own data, without making things up

• What is RAG and what problem does it solve?
• How does the RAG pipeline work in practice?
• Why do chunking and embeddings decide quality?
• What goes wrong in RAG and how do you fix it?
• How do you evaluate a RAG system?
• RAG, fine-tuning or long context: when to use each?

A language model does not know your company's documents and has no way to cite a source for what it claims. RAG solves both problems with a simple idea: retrieve the right passages before asking, and require the answer to rest on them.

What is RAG and what problem does it solve?

RAG stands for Retrieval-Augmented Generation. The name describes the mechanism: before the model answers, a search system retrieves from your base the passages most relevant to the question and injects them into the prompt. The model then answers while looking at that material.

The architecture was formalized in a Meta AI paper in 2020 and became the corporate standard because it solves three language model limitations at once:

1. No knowledge of what is yours. No model was trained on your internal handbook, your client contracts or your ticket history.
2. A knowledge cutoff. Training has an end date. Everything that happened afterward is invisible to the model.
3. No source. A bare LLM asserts things without showing where it got them. In legal, medical or financial contexts, that makes it unusable.

The obvious alternative — putting all the documents into the prompt — runs into cost and quality. Even with large context windows, sending thousands of pages with every question is expensive in tokens and degrades accuracy, because the relevant information gets diluted in noise.

Practical example: asking a generic model "what is the warranty period in contract 4471?" produces a plausible, invented answer. With RAG, the system retrieves the exact clause from that contract and the answer arrives with a citation — verifiable in seconds.

How does the RAG pipeline work in practice?

The system has two halves that run at different times.

Indexing (once, and on every update):

1. Extraction. Converting PDFs, spreadsheets, pages and databases into clean text. A tedious and decisive step: a badly extracted table becomes a wrong answer.
2. Chunking. Splitting the text into pieces. This is the single most impactful decision in the whole pipeline.
3. Embeddings. Turning each piece into a vector using an embedding model.
4. Storage. Keeping vectors and metadata — source, date, permission, section — in a searchable index.

Querying (on every question):

1. Retrieval. Generating the question's embedding and retrieving the nearest passages.
2. Reranking. Re-evaluating the candidates with a more precise model and keeping the best ones.
3. Prompt assembly. Inserting the chosen passages along with the instruction and the question.
4. Generation. The model answers, citing sources.

def answer(question: str) -> dict:
    query = embed(question)
    candidates = index.search(query, k=20, filters={"permission": user.groups})
    chunks = rerank(question, candidates)[:5]

    context = "\n\n".join(
        f"[{c.id}] (source: {c.source}, {c.date})\n{c.text}" for c in chunks
    )
    instruction = (
        "Answer using ONLY the context below. Cite sources in square brackets. "
        "If the context does not contain the answer, say you could not find it.\n\n"
        f"{context}\n\nQuestion: {question}"
    )
    return {"answer": llm(instruction), "sources": [c.source for c in chunks]}

Notice the permission filter inside the retrieval call. It is the detail that separates a prototype from a corporate system: access control has to happen at retrieval, not at generation. Trusting the model "not to tell" what it already received in context is a poor security decision.

Practical example: an HR assistant indexed documents from every department and applied the access rule only in the prompt instruction. One indirect question was enough for data from another area to leak into the answer. The fix was filtering by group in the index, before retrieval.

Why do chunking and embeddings decide quality?

When a RAG system answers badly, the cause is almost never the language model — it is retrieval bringing back the wrong passage. And retrieval depends on two choices.

Chunking is how the text is cut. Chunks that are too large bring noise along with signal; too small and they lose context and return sentences with no meaning. Three strategies cover most cases:

The rule of thumb: respect the structure the document already has. A contract has clauses, a regulation has articles, a manual has sections. Cutting every 500 tokens while ignoring that splits information in half and forces retrieval to guess.

Embeddings define what "similar" means. The concept is the same as any similarity measure between vectors — the intuition behind Euclidean distance applied across hundreds of dimensions, except that cosine similarity is normally used here. Three things matter: picking a model that performs well in your language, checking the embedding model's token limit and — a non-negotiable rule — using the same model for indexing and for querying.

A fourth decision prevents the most frustrating failure: adopt hybrid search, combining vector similarity with keyword search. Vectors capture meaning but fail on codes, acronyms and numbers — exactly what people search for most in a corporate base.

Practical example: a ticket base returned poor results for queries like "error 4711". Vector search understood "error" and ignored the number. With hybrid search, the code gained weight and accuracy went up immediately.

What goes wrong in RAG and how do you fix it?

Five failures account for most problems in production systems:

1. The right passage was never retrieved. Nothing you do in the prompt fixes this. Investigate chunking, consider hybrid search and retrieve more candidates before reranking.
2. The right passage was retrieved and the model ignored it. This usually happens when the context is too long and the relevant information sits in the middle. Reduce the number of passages and order them by descending relevance.
3. The answer goes beyond the source. Explicitly instruct it to answer only from context, require citations and allow "I could not find it". Models prefer answering to admitting absence — you have to authorize the opposite.
4. The base ages silently. An updated document with a stale index produces confident, wrong answers. Reindexing has to be part of the data pipeline, with the update date visible in the answer.
5. Questions the architecture cannot serve. RAG retrieves passages; it does not sum, count or compare across the whole base. "How many contracts expire this month?" is a database query, not a similarity search. The way out is giving the agent a structured query tool alongside the RAG.

Practical example: a legal assistant got conceptual questions right and systematically failed counting questions. The fix was not improving the RAG — it was adding a SQL query tool and letting the model choose which one to use based on the question.

How do you evaluate a RAG system?

Without measurement, you have a convincing demo and an unpredictable product. Evaluation has to separate the two halves:

Retrieval — assemble a set of 50 to 100 real questions with the correct passage annotated by hand. Measure how often the right passage appears among those retrieved and where it typically ranks. That set is the project's most valuable asset: it lets you swap the embedding model, change chunking or enable reranking and know, with numbers, whether it improved.

Generation — over the retrieved passages, evaluate three dimensions: faithfulness (is the answer contained in the context?), relevance (does it answer what was asked?) and coverage (did it use everything it needed?). Tools like Ragas automate part of that using a model as judge, which is useful for tracking trends — but periodic human review remains necessary.

Two operational metrics complete the picture: latency per stage, to know where time is lost, and cost per answer, which tends to surprise when reranking uses an expensive model.

Practical example: a team swapped the embedding model expecting everything to improve, and satisfaction dropped. The evaluation set showed why: the new model was better in English and worse in Portuguese, and retrieval fell eight points. Without the set, the conclusion would have been "the language model got worse".

RAG, fine-tuning or long context: when to use each?

The three approaches solve different problems and are frequently confused:

The choice becomes simple with one question: is what you want to teach the model a fact or a behavior? A fact — a refund policy, a clause, a specification — is RAG. A behavior — always answer in report format, use the house terminology — is fine-tuning. And if the material fits entirely in the context and changes with every conversation, sending it directly is simpler than building a pipeline.

Note that the three combine. A mature system might have a model fine-tuned for the company's format, RAG for the facts and a large context window for the document the user just attached. And when that retrieval needs to be offered to several different AI applications, the Model Context Protocol has become the standard way to expose the base as a reusable tool.

Practical example: an insurer tried fine-tuning to teach coverage rules to the model. Every policy change required retraining, and the model still could not cite a source. Moving to RAG solved both problems — and fine-tuning stayed, but only for the standardized format of its reports.

Conclusion

RAG is the most direct way to make a language model work with knowledge that is yours, without retraining anything and with answers anchored in a source you can check. The architecture is simple to describe and demanding to execute: almost all the outcome depends on unglamorous decisions — how the document is extracted, how the text is cut, which embedding model understands your language, whether retrieval combines semantics and keywords, and whether access control happens at retrieval. When one of these systems disappoints, investigate retrieval before swapping the model: in the overwhelming majority of cases, the right passage never reached the prompt. And before any optimization, build the evaluation question set — it is what turns opinions about quality into a comparable number, and what lets you genuinely improve instead of swapping parts in the dark.