Should I use RAG or fine-tuning for a chatbot that answers questions about my documentation?

RAG, in almost every case. Documentation changes, and RAG lets the model cite the current version without retraining. Fine-tuning would bake a snapshot of the docs into the weights, and that snapshot goes stale the next time you edit a page.

Can I combine fine-tuning and RAG?

Yes, and it is common. Fine-tune the model to follow your output format, tone, or tool-calling conventions, then use RAG to feed it current facts at inference time. The two changes are independent and address different failure modes.

Does fine-tuning teach a model new facts reliably?

Not reliably, and it is one of the more persistent misconceptions about fine-tuning. Training on a fact a handful of times nudges the model's weights toward it but does not guarantee recall the way a retrieved document in context does, and there is no way to verify after the fact that a specific fact was learned versus only partially absorbed.

Decision guide

Blog/Fine-tuning vs RAG: how to actually choose

For AI assistants

Fine-tuning vs RAG: how to actually choose

By Samuel Seidel · September 9, 2026

Teams usually arrive at this question already leaning toward fine-tuning, because it sounds like the more serious engineering answer. Most of the time it's the wrong one. RAG and fine-tuning fix different kinds of problems, and picking the wrong one means paying for training runs that don't move the number you actually care about.

What each one actually does

Retrieval-augmented generation looks up relevant text at the moment of the request and puts it into the prompt, so the model answers with your current documents in front of it rather than from what it memorized during training. The model's weights never change. Swap out a document, and the next query sees the update immediately.

Fine-tuning changes the weights. You show the model thousands of examples of the input-output behavior you want, and gradient updates shift it toward producing that behavior by default, without needing the instruction spelled out in the prompt each time. The model comes out of the process different: not smarter about your documents, but different in how it responds.

That distinction, updating what the model knows at request time versus updating how the model behaves permanently, is the entire decision. Almost every mistake in this space comes from treating them as interchangeable ways to "add knowledge."

Reach for RAG when the knowledge changes or needs a citation

If your answer depends on something that can be different next week, a price, a policy, an inventory count, a support ticket status, retrieval is the only approach that keeps up without retraining. You update the source document and the model's next answer reflects it. There is no training run, no evaluation cycle, no waiting for a new checkpoint to catch up with reality.

RAG also gives you something fine-tuning structurally can't: a pointer back to the source. When the model answers from retrieved context, you can show which document it pulled from, which paragraph, which page. That matters more than it sounds like it should the first time a customer or auditor asks "where did that come from" and the honest answer for a fine-tuned fact is "somewhere in the training data, we can't say exactly where."

The traceability point alone rules fine-tuning out for a lot of regulated or customer-facing use cases. If an answer needs a citation, or needs to be provably wrong when the underlying document turns out to be wrong, RAG is cheaper, and it's the only approach that supports the requirement at all.

Reach for fine-tuning when the problem is behavior, not knowledge

Fine-tuning earns its cost in a narrower set of situations: you want the model to consistently produce a specific output format (a particular JSON schema, a house style, a specific tone), follow a domain-specific reasoning pattern, or call tools in a way that's idiosyncratic to your system. These are behavior problems. No amount of retrieved context fixes a model that keeps drifting out of your required schema, because the instruction "always respond in this exact format" is competing against everything else in a long system prompt, and a long enough conversation eventually erodes it. Training the behavior in makes it the default rather than a request the model has to remember to honor.

Fine-tuning also fits genuinely stable domain knowledge, terminology, conventions, and patterns that aren't going to be edited next quarter. A model trained to understand a specialized vocabulary or a consistent internal shorthand doesn't need that context re-injected on every call, which matters once you're paying for tokens in the prompt on every request at scale.

Our own guide to fine-tuning on your data covers the mechanics if this is the path you're on: what a training run actually needs in terms of data volume and hardware, and where the process is more forgiving than people expect.

The common mistake: fine-tuning to fix a knowledge problem

The failure pattern we see most often looks like this: a team notices the model gives wrong or outdated answers about their product, concludes the model "doesn't know enough" about their business, and starts collecting Q&A pairs for a fine-tuning dataset. Weeks later, the fine-tuned model still gets facts wrong, because fine-tuning was never a reliable way to teach specific facts in the first place. Training on a fact a few dozen times nudges the model's weights in that direction; it doesn't guarantee the model will retrieve that exact fact correctly under a different phrasing of the question, and there's no way to audit afterward whether it actually stuck.

Worse, the underlying problem, that the documentation changed since the model was trained, doesn't go away. The team is now stuck re-running a fine-tune every time a product spec updates, which is slower and more expensive than editing a document and re-indexing it, which is what RAG would have needed instead.

If the actual complaint is "the model doesn't know about X," ask whether X is written down somewhere. If it is, that's a retrieval problem, not a training problem, and RAG will fix it faster, more cheaply, and more verifiably than a fine-tune ever will.

A short framework

Ask, in order: Does the answer depend on information that changes? If yes, RAG. Does the answer need to be traceable to a source? If yes, RAG. Is the problem that the model gives the right facts in the wrong format, tone, or structure? If yes, fine-tuning. Is the problem that the model doesn't know something written down somewhere in your documents? If yes, that's RAG wearing a fine-tuning costume, and RAG is what you actually need.

Most production systems that need both end up combining them: fine-tune for format and behavior, RAG for facts. The two are not competing techniques so much as two knobs that happen to get confused for each other early in a project, before anyone has separated "the model doesn't know this" from "the model won't say it the way I want."