RAG or Fine-Tuning: Five Questions That Decide It Before You Spend

Someone on the team has proposed fine-tuning a model on the company's documents. Someone else says retrieval would do the same job for less. Both are describing a real technique, and the two techniques solve different problems. The decision turns on which problem you have, and five questions settle it in an afternoon.

The short version first. Use RAG when the job is factual, which covers most enterprise knowledge work. Fine-tune when the gap is how the model reasons or writes, and you can produce the examples to teach it. When in doubt, start with RAG. It is cheaper to build, updates in minutes, and cites its sources by design.

What each one changes

RAG (retrieval-augmented generation) keeps your knowledge outside the model. At query time the system searches your documents, pulls the relevant passages, and gives them to the model as context before it writes. The model never changes. The design comes from Lewis et al. (2020), who separated retrieval from generation on purpose, so that the knowledge could be updated without touching the model.

Fine-tuning continues training a model on your examples until its weights shift. The model changes permanently. What it learns is patterns: how to reason about a kind of problem, how to phrase things, what format to produce. It is poor at memorizing facts reliably, and anything it does memorize is frozen at training time.

Keep one sentence in mind through the five questions. Retrieval changes what the model knows at the moment of the question. Fine-tuning changes how it behaves on every question.

The five questions

Ask them in order. The first one with a clear answer usually decides it.

1. Does the knowledge change more than once a month?

Product catalogs, price lists, case law, clinical protocols and policy manuals all do. Re-indexing a changed document in a RAG system takes minutes and costs almost nothing. Keeping a fine-tuned model current means another training run each time, with another round of example preparation before it.

The retraining cost is a formula you can fill in yourself.

Annual retraining compute = GPU hours per run × hourly GPU rate × runs per year.

As an illustration only, take Lambda's published rate of $4.29 an hour for a single H100 SXM. A run that takes 8 GPU hours, repeated monthly, comes to 8 × $4.29 × 12, or about $412 a year in compute. The compute is the small part. The large part is the people who prepare and check the examples before every run, and that cost repeats twelve times too.

If the knowledge changes often, RAG wins on operations alone.

2. Do users need to see where the answer came from?

A clinician asking for a dose, a lawyer asking for a clause, an auditor asking what the policy said in March. Each needs the source, not just the answer. RAG returns the passages it used, so a citation to a page is native to the design. A fine-tuned model has blended its training data into weights and cannot point to a source. It produces fluent answers with nothing to check.

For legal and clinically sensitive material, this question alone usually decides it.

3. Must the documents stay inside your infrastructure?

A RAG system keeps the documents in your own index: a self-hosted vector database behind your firewall. The model sees only the passages retrieved for one question, and on-premise it sees nothing that leaves the building. A private build on a single GPU shows how small that footprint can be.

Fine-tuning sends the training set to whoever runs the training job. With a hosted provider, every example is written to their storage. With your own cluster, you carry the GPU cost and the operations. For privileged, clinically sensitive or commercially sensitive material, this question often rules fine-tuning out before anyone discusses cost.

4. What kind of questions do users ask?

Sort the questions people ask into four kinds.

Questions that span several documents and a live system ("which active clients have contracts missing the new addendum?") are a fifth kind. They need retrieval that plans more than one search, which is agentic RAG, not fine-tuning.

The line that matters: fine-tuning improves patterns, not recall. If users ask what a document says, fine-tuning does not help. If users need the model to reason like a senior person in your field, it can.

5. Can you produce the training examples?

Fine-tuning on thin or noisy data produces a model that fails in ways nobody can debug. The vendors publish their own floors, and they are lower than the folklore. OpenAI's supervised fine-tuning guide sets the minimum at 10 examples and says improvements usually show at 50 to 100. Its advice when 50 examples have no impact is to rethink the task or the prompt before adding training data. Anthropic's fine-tuning for its small Claude model runs through Amazon Bedrock. Its data preparation page sets a minimum of 32 records and a maximum of 10,000 training records.

Those are vendor minimums for the job to run, not the number that produces a model you would trust with customers. The examples have to be correct, consistent, representative of real traffic, and reviewed by someone who knows the domain. Producing a few hundred of those is weeks of expert time. Producing a few thousand is a project. RAG needs none of this. It needs the documents you already have.

What getting it wrong costs

Pick fine-tuning for a factual job and you get a model that is confidently out of date within a month. There are no citations to catch it, and there is a retraining bill every time the documents change. Pick RAG for a style job and you get accurate answers in the wrong voice, with prompt instructions doing work that weights should do. The first mistake is more expensive and more common, because fine-tuning sounds like the more serious investment. It is only the more specific one.

There is a third option that both camps skip. Few-shot prompting puts a handful of worked examples in the prompt itself. For reasoning and format, it closes much of the gap at no training cost. It is the test to run before any fine-tuning budget is approved. If ten good examples in the prompt do not move the result, fifty in a training set rarely will.

The hybrid, and why it comes second

Mature systems sometimes run both: retrieval for the facts, and a fine-tuned model for how it reasons about them. That is the most expensive design, and the right one for a narrow set of problems. It usually arrives after a RAG system is live and its test set shows a reasoning gap that retrieval cannot close. The order matters. Build retrieval first, measure, and fine-tune against a named, measured gap.

Where fine-tuning wins

Four cases, all narrow.

Consistent voice and format. Customer-facing copy, regulated document templates, a rigid JSON schema the application depends on. Retrieved context changes what the model says, not how it says it.

Structured output and tool calls at volume. If every output follows one schema, training it in shortens prompts and cuts failures more than retrieval can.

Latency where retrieval cannot fit. A fine-tuned model answers from weights with no search step. That is rare in knowledge work and real in some product features. Measure your retrieval step before assuming it is the bottleneck.

A large, curated example set already exists. Call-center transcripts with graded outcomes, or a legal team's structured case database. Then the fine-tuning cost is the compute, and the formula above makes it small.

Frequently asked questions

Can RAG hallucinate? Yes, in its own way. Without a no-answer rule and an output check, a model can still invent beyond the retrieved text. Production systems add both, and the rate of unsupported claims is measured on a fixed test set before launch rather than assumed.

How does RAG handle very long documents? With chunking that respects the document's structure. Keep sections whole, extract tables separately, and pair a small chunk for search with its parent section for the model. Scans, charts and mixed layouts are a multimodal pipeline's problem.

What vector database should I use? pgvector if your vectors live beside relational data in Postgres. Qdrant for heavy filtering or in-country hosting. Pinecone if nobody will run infrastructure. Our vector database comparison works through the memory arithmetic at 10 million vectors.

Does RAG work with tables and spreadsheets? Partly. Tables can be embedded as text and retrieved. Numeric questions are better served by a SQL layer, with retrieval reserved for prose. Detect the intent, then send each question to the right backend.

Before the next planning meeting, write down the ten questions your users most ask of the system you are planning. Mark each one fact, synthesis, reasoning or style, and count. If fact and synthesis dominate, build RAG and measure it. Fine-tuning is only on the table if reasoning and style dominate and you can name who will write and check the examples.