Fine-Tuning vs RAG

Build the mental model

RAG and fine-tuning solve different problems and are frequently reached for interchangeably by mistake.

RAG gives a model external knowledge at request time: the weights never move, and a new document is usable the moment it's indexed.

Fine-tuning changes the model's actual weights through additional training on examples, which means the model 'remembers' what it learned without needing that information handed to it in the prompt again — but only after a training run that costs real compute time and, for anything beyond small adapter-based methods, real GPU memory.

The practical comparison is stark.

Dimension	RAG vs Fine-tuning
Knowledge update	RAG is instant (re-index one document); fine-tuning means retraining, evaluating, and redeploying a new model checkpoint.
Cost and hardware	RAG needs an embedding model and a vector store, both cheap and often runnable on CPU; full fine-tuning needs a GPU with meaningfully more VRAM than inference alone, for hours.
Complexity	RAG is a data-pipeline problem; fine-tuning is a training problem, with its own risks like overfitting or catastrophic forgetting of prior capability.
ဘယ်ဟာက ဘာမှာ ကောင်းလဲ	RAG is strong at adding or updating facts, weak at changing how the model talks or reasons; fine-tuning is the opposite — strong at durable shifts in style, tone, format-following, or domain-specific behavior, weak as a way to keep facts current since every fact baked into weights goes stale the moment reality changes again.

The lighter-weight middle ground worth knowing about is LoRA and QLoRA — adapter-based fine-tuning that trains a small number of additional parameters instead of the full model, needing a fraction of the compute and memory of full fine-tuning, without requiring you to actually run a training job in this lesson to understand the tradeoff it represents.

Fine-tuning: Additional training performed on a pretrained model using a task- or domain-specific dataset, which updates the model's weights and produces a new checkpoint — distinct from RAG, where the base weights never change.

LoRA: Low-Rank Adaptation — a fine-tuning technique that trains a small number of additional low-rank weight matrices instead of updating the full model, dramatically reducing the compute and memory needed compared to full fine-tuning.

text

RAG (REQUEST-TIME) VS FINE-TUNING (TRAINING-TIME)
-------------------------------------------------
RAG (REQUEST-TIME) VS FINE-TUNING (TRAINING-TIME)
------------------------------------------------------

RAG                                FINE-TUNING
---                                -----------
question                           training examples
   |                                    |
   v                                    v
retrieve context                   gradient descent
   |                                updates model weights
   v                                    |
context + question -> LLM               v
   |                                new model checkpoint
   v                                    |
answer                                  v
                                    question -> LLM(new) -> answer

weights: unchanged                 weights: changed
update speed: instant              update speed: hours+ per run

Connect it to a real scenario

The code example shows the mechanical core of what 'RAG at request time' actually means: a plain string-formatting function, no training loop anywhere. Given a list of already-retrieved chunks and a question, it concatenates them into a single prompt string with an instruction telling the model to answer only from the provided context — that assembled string is the only thing that changes between requests.

Compare this to what a fine-tuning run would require instead:

A dataset of examples (potentially thousands)
A training loop that updates weights over many passes through that data
A held-out evaluation set to check the fine-tuned model didn't regress on unrelated tasks
A new model checkpoint to deploy and version afterward

This asymmetry is the practical argument for defaulting to RAG whenever the actual need is 'the model should know X,' where X changes: prices, inventory, today's schedule, this week's documentation.

Reach for fine-tuning instead when the need is behavioral and durable — a consistent output format, a specific tone across all responses, following a domain-specific convention that's hard to state fully in a system prompt every time.

LoRA and QLoRA narrow fine-tuning's cost gap by training small adapter weights instead of the full model, but they still require an actual training run, unlike RAG.

RAG injects knowledge into the prompt at request time with unchanged weights; fine-tuning changes the weights themselves through additional training.
Prefer RAG for facts that change; consider fine-tuning for durable shifts in style, tone, or domain-specific behavior that are hard to restate in every prompt.
LoRA/QLoRA are adapter-based fine-tuning methods that need far less compute than full fine-tuning, but they still require an actual training run, unlike RAG's instant updates.

Try the working example

python

def build_rag_prompt(retrieved_chunks, question):
    context = "\n".join(f"- {c}" for c in retrieved_chunks)
    prompt = (
        "Answer the question using only the context below. "
        "If the answer is not in the context, say you don't know.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\n"
        "Answer:"
    )
    return prompt


retrieved_chunks = [
    "The Basic Plan costs $9/month and includes 5 projects.",
    "The Pro Plan costs $29/month and includes unlimited projects.",
]
question = "How much does the Pro Plan cost?"

final_prompt = build_rag_prompt(retrieved_chunks, question)
print(final_prompt)
print("---")
print(f"prompt length: {len(final_prompt)} chars")

You should see

build_rag_prompt with the two pricing chunks and the pricing question produces:

Answer the question using only the context below. If the answer is not in the context, say you don't know.

Context:
- The Basic Plan costs $9/month and includes 5 projects.
- The Pro Plan costs $29/month and includes unlimited projects.

Question: How much does the Pro Plan cost?
Answer:
---
prompt length: 289 chars

Everything above 'Answer:' is what actually gets sent to the LLM as a single prompt string — there is no separate 'training' artifact produced anywhere in this process, only string concatenation.

5-minute try-it

Change retrieved_chunks to an empty list [] and re-run build_rag_prompt — consider why the 'say you don't know' instruction matters especially when there's no context to send to the LLM. Then modify the function to also accept a source_name per chunk, formatting each context line as [Source: filename] so the assembled prompt shows which document each chunk came from.

One important caution

Fine-tuning a model to 'teach it' a fact that changes regularly (prices, stock, today's date) — the fact goes stale the moment reality changes again, and RAG solves this at a fraction of the cost.

Expecting RAG alone to fix inconsistent tone or formatting — retrieval adds facts to the context, it doesn't reliably change how the model expresses itself across every response.

Quick Check

You need the model to always answer using this week's product prices, which change every few days. What's the better first choice?

A. Fine-tune the model weekly on the new price list.B. Use RAG — index the current price sheet and retrieve it at request time; no retraining needed as prices change.C. Fine-tune once on historical prices and let the model estimate current ones.D. Neither — this can't be solved without a cloud model.

Hugging Face PEFT Documentation — Local AI / Local LLM