When a business decides to add AI to its product — a smarter search, a question-answering feature over its documents, a customer support assistant — it quickly runs into a fundamental question: should we use retrieval-augmented generation (RAG), or fine-tune a model? The answer depends on the problem you are solving, and getting it wrong wastes months and a significant budget.
What RAG does
Retrieval-augmented generation combines a language model with a search step. When a user asks a question, the system first retrieves relevant passages from your own knowledge base — documentation, policies, product data, past tickets — and passes them to the model together with the question. The model then answers based on that material rather than only on what it learned in training.
A RAG system has four main parts:
- an embedding model that turns documents and questions into vectors;
- a vector store — pgvector inside PostgreSQL, Qdrant, Weaviate, or Pinecone;
- a retrieval step that finds the most relevant passages, often combined with traditional keyword search;
- a prompt that tells the model to answer from the retrieved context and to say so when the answer is not there.
When RAG is the right choice
RAG fits when the information changes often, is private to your business, or was never part of the model's training data. Support assistants that answer from your product documentation, internal Q&A over company wikis, and research tools over contracts or regulations are typical examples.
Its biggest practical advantage: you update the knowledge base, not the model. A new price list or policy is live as soon as it is indexed. Answers can also cite their sources, which builds trust with users and makes errors easy to trace.
What fine-tuning does
Fine-tuning continues training a model on examples from your use case. The result is a model that has absorbed a particular style, format, vocabulary, or way of handling a task. It does not look anything up — the behaviour is built into the model's weights.
When fine-tuning is the right choice
Fine-tuning fits when you need:
- a consistent output format or style that prompting alone does not achieve reliably, such as a strict structured output or a brand's tone of voice;
- a specialised task where you have thousands of good input-output examples and the base model clearly underperforms;
- lower cost or latency at high volume, by teaching a smaller model to do one task as well as a larger general model.
Fine-tuning is usually the wrong tool for *knowledge* that changes. Updating what a fine-tuned model knows means preparing new data and training again. Getting it right also needs clean, well-labelled examples and a proper evaluation setup.
The hybrid approach
Many production systems use both: a model fine-tuned for format, tone, and domain vocabulary, combined with retrieval that supplies current facts at the moment of the question. You get consistent, well-structured answers that are still grounded in up-to-date information.
How to decide for your application
- Start with good prompting. Modern models do a lot with clear instructions and a few examples.
- Add RAG when the answers depend on your own or changing information.
- Consider fine-tuning only when you have a measurable gap — a format, style, or task the model still gets wrong — and enough quality data to fix it.
Whichever route you take, measure before you decide. Build a test set of at least 100 representative questions with expected answers and score each change against it. RAG quality depends heavily on how documents are split into chunks, which embedding model you use, and how many passages you retrieve; tune those before concluding that fine-tuning is needed.
Cost and maintenance
RAG adds retrieval infrastructure and longer prompts (the retrieved passages cost tokens on every request), but it is cheap to update. Fine-tuning has an upfront training cost and needs retraining when requirements change, but it can cut per-request costs when a small fine-tuned model replaces a large one. Our article on the hidden costs of running AI in production covers the running costs in more detail.
If you are planning an AI feature and want a second opinion on the architecture, our team can review your use case and recommend the simplest approach that will work. See our AI & machine learning services or get in touch.



