When a team wants a language model to work with their own information, the question soon comes up: should we use retrieval (RAG) or fine-tune a model? The two are often presented as alternatives. In practice they solve different problems, and many production systems use both.
What each one does
Retrieval augmented generation (RAG) gives the model the right information at the moment it answers. When a question comes in, the system searches your documents, tickets or records, picks the most relevant passages and passes them to the model with the question. The model's own knowledge is unchanged. It is simply given better material to work from.
Fine-tuning changes the model itself. You train it further on examples of the behaviour you want, so it learns a style, a format or a narrow task more reliably.
A useful shorthand: RAG changes what the model knows right now. Fine-tuning changes how the model behaves.
When RAG is the right starting point
Most business use cases start here. RAG is usually the better fit when:
- Your content changes. Policies, prices, products and procedures are updated. With RAG, you update the source and the answers follow.
- Answers need sources. People need to check where an answer came from, and auditors may need to as well.
- Access rules apply. Different people may see different documents. Retrieval can respect those permissions. A fine-tuned model cannot forget what it was trained on.
- You need to start quickly. RAG needs no training data set, just well organised content.
When fine-tuning helps
Fine-tuning earns its place when:
- The format must be exact. Structured outputs, specific report layouts or a strict house style.
- The task is narrow and repetitive. Classification, extraction or routing where you have many good examples.
- Cost or speed matter at scale. A smaller fine-tuned model can sometimes match a larger general model on one task, at lower cost and latency.
- The language is specialised. Domains with their own vocabulary, where a general model struggles even with good context.
When to combine them
A common production pattern uses both: retrieval supplies current, permissioned facts, and a fine-tuned or carefully prompted model turns them into the right output. For example, an assistant that drafts responses to customer claims might retrieve the policy and the claim history, then use a model tuned on approved past responses to write the draft in the right tone and structure.
How to decide with evidence
The decision should come from testing, not preference. A simple approach:
- Collect real examples of questions or tasks, with what a good answer looks like.
- Build a baseline with retrieval and good prompts. It is faster to build and often good enough.
- Measure it against the examples for accuracy, safety, cost and latency.
- Fine-tune only where the baseline falls short on something fine-tuning can fix, such as format, style or a narrow task.
- Keep measuring in production, because content, users and models all change.
A note on data and ownership
Whichever route you take, it matters where your data goes. Retrieval indexes and fine-tuned models should live in your own environment, under your access and retention rules, with providers and settings that do not train on your data. And the evaluation set you build along the way is an asset in its own right: it lets you compare models and approaches as the options keep changing.
If you are weighing this choice for a specific use case, we can help you test it. Book a discovery call.