RAG or Fine-Tuning: What to Choose for Your Knowledge Base

The request “fine-tune the model on our documents” sounds logical: we have knowledge, the model does not, let’s put one into the other. The problem is that fine-tuning does not do what people expect. It does not “load knowledge” into the model — it shifts the distribution of its answers. Let’s break down what actually happens in both approaches, where each breaks, and how to decide based on numbers rather than faith.

Two approaches — two different mechanisms

RAG (Retrieval-Augmented Generation) separates knowledge from reasoning. Documents live in an external index — usually hybrid: vector search over embeddings plus BM25 by keywords. For a query the system retrieves 5–15 relevant fragments and feeds them to the model in the context with an instruction to answer only based on them. The model here is an engine of understanding and formulation, not a storage of facts.

Fine-tuning — continued training on your examples of “input → desired output”. In corporate practice this is almost always LoRA/QLoRA on top of an open model (Qwen, Llama, Mistral) — full weight fine-tuning makes no economic sense. Result: the model starts writing like your examples — in the right format, tone and structure. It memorizes facts from examples unreliably and reproduces them with distortions; this is confirmed both by research and by our measurements — on closed questions from regulations, a fine-tuned 7B model without RAG gives 40–55% accuracy versus 85–90% for RAG on the same base.

Why RAG wins in 9 out of 10 enterprise tasks

• Freshness. The price list or a regulation changed — we updated one file in the index. A fine-tuned model would have to be retrained.
• Transparency. Every answer references a specific source paragraph. Lawyers and security teams love that.
• Access control. A manager sees answers only for their own documents — a filter at the search level, not “we asked the model to stay silent”.
• Price. Indexing 10,000 pages is hours of work and kopecks in tokens. Fine-tuning is weeks, GPUs and a dataset you still have to collect.

Where RAG breaks — and it’s not about the model

Honestly: “tutorial-grade” RAG systems work poorly, and disappointment in them is what often pushes people toward fine-tuning. Three places where everything breaks in practice.

Chunking. Cutting documents into 512-token chunks with overlap is a way to get a pricing table split into three fragments and a contract clause losing its section heading. We cut along the document structure (headings, clauses, table rows), add “breadcrumbs” — the path from the document root — to each chunk, and for tables we keep the whole row with the header. This gives more accuracy gain than switching models.

Retrieval. Pure vector search poorly catches part numbers, order IDs, abbreviations — everything that makes up half of queries in corporate language. A hybrid with BM25 and a reranker (bge-reranker or Cohere Rerank if data may go outside) closes this. We measure recall@k separately from answer quality: if the right fragment did not make it into the top 10, the model is not to blame.

Conflicts in the sources. A corporate knowledge base has three versions of a regulation, two of which are outdated. RAG will honestly find all three. Fixed by metadata (date, document status), filtering at the search stage, and — alas — cleaning up the knowledge base that has been postponed for five years.

When fine-tuning is justified

• A rigid output format. Filling structured forms, generating code in an internal DSL, labeling under a complex 40-class scheme. Prompting with examples gives 80%, LoRA on 2–3 thousand examples — 95%+.
• Domain language. Medical discharge summaries, oilfield service reports, legal opinions in the style of a specific practice. The base model writes “correctly, but not the way we do it”.
• Inference economics. Millions of requests a month, where a fine-tuned 7–8B model replaces a 70B or an external API. The cost difference is an order of magnitude; it pays back within a quarter.
• Latency. Voice agents, where a 3–4 second RAG cycle is unacceptable and a small specialized model answers in 400 ms.

In all four cases fine-tuning goes on top of RAG: facts still come from the index, and the model is fine-tuned to present them. The “fine-tuning only” scheme without retrieval has never survived in our production.

How to decide: the protocol

We run it on every pilot, and it takes a week, not a quarter.

1. 1We collect 200–300 real questions from the logs and label reference answers with the client’s expert. Half we set aside.
2. 2We measure search recall@10 separately. Below 90% — we fix chunking and retrieval and do not touch the model.
3. 3We run RAG on a strong model (Claude/OpenAI or Qwen 72B in the perimeter). Accuracy above the target — fine-tuning is not needed, case closed.
4. 4We classify the errors. If “found but formulated wrong” dominates — that is a LoRA candidate. If “not found” — we go back to step 2.
5. 5We count the economics: the cost of collecting the dataset and training versus the difference in inference cost over 12 months. Fine-tuning wins only at high volume or rigid format requirements.

The takeaway for a CTO: if you are offered “fine-tune a model on the documents” as the first step — that is a red flag. The right order is retrieval, prompting, evaluation, and only then, with proven necessity, weights.