Fine-tuning or RAG? The question is what you are missing

The most common expensive mistake I see is fine-tuning a model on a company's documentation and expecting it to answer questions about that documentation. It will not, reliably, and the reason is worth understanding before you spend a week on it.

Two different gaps

Fine-tuning adjusts weights. It is how you change how a model behaves — its format, its tone, its willingness to follow an unusual instruction shape, its handling of a specialised vocabulary. It is pattern absorption, not memorisation, and it does not give you a reliable lookup table.

Retrieval puts text in the context window. It is how you supply what the model does not know — this quarter's numbers, this client's contract, the policy that changed last week. It is exact, it is auditable, and it updates the moment the document does.

Why fine-tuning on documents disappoints

Train on a policy handbook and the model learns the style and vocabulary of policy handbooks. Ask it a specific question and it produces something that reads exactly like your handbook — fluent, correctly formatted, plausibly worded, and not necessarily what the handbook says. You have taught it the shape of the answer, not the answer.

Worse, there is no citation to check and no way to correct it short of retraining. A retrieval system given the same question returns the paragraph, and a wrong answer is visibly wrong because the source sits next to it.

When fine-tuning is genuinely right

• You need a rigid output format that prompting keeps drifting away from across long conversations.
• Your domain language is far enough from ordinary English that the base model misreads terms — clinical notes, legal drafting, a specific transliteration convention.
• You want a much smaller model to imitate a much larger one on one narrow task, to cut cost or latency by an order of magnitude.
• The behaviour you want takes 2,000 tokens of instructions to describe, and you are paying for those tokens on every single call.

That last case is the one people underrate. If your prompt has grown to a page of rules and examples, fine-tuning the rules into the weights can be cheaper per request and more consistent than re-explaining them forever.

LoRA, and why you do not need a datacentre

LoRA freezes the base weights and trains small low-rank adapter matrices alongside them. You end up training a fraction of a percent of the parameters, the adapter is a few megabytes rather than gigabytes, and you can hold several task-specific adapters against one base model. QLoRA quantises the frozen base to 4-bit as well, which is what brings a 7B fine-tune onto a single consumer GPU.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

# 4-bit base: this is the difference between one GPU and several.
quantisation = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

base = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.2",
    quantization_config=quantisation,
    device_map="auto",
)

# Target the attention projections. r=16 is a sane starting point;
# raise it only if the model is visibly underfitting the task.
config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)

model = get_peft_model(base, config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 3,765,776,384 || trainable%: 0.36

In practice, both

The systems I am happiest with retrieve the facts and fine-tune the behaviour. Retrieval keeps answers current and citable; the adapter keeps the response format and domain register stable without a page of instructions on every call. They solve different problems and compose cleanly.

1. Start with prompting alone. It is free and it settles what "good" looks like.
2. Add retrieval as soon as answers depend on facts the model cannot have.
3. Only once both are in place, ask whether the remaining failures are behavioural.
4. If they are, fine-tune on the smallest high-quality dataset that covers them.
5. Keep the evaluation set from step one. It is the only way to know the adapter helped.

Most projects never reach step four, and that is the correct outcome. Fine-tuning is a real tool with a real cost, and the question that matters is not which technique is better — it is which gap you are actually looking at.