RAG vs. fine-tuning vs. long context: how to actually decide

The question everyone asks wrong

“Should we fine-tune or use RAG” treats these as competing options for the same job. They’re not. Each does something structurally different, and the right production system usually uses more than one of them for different reasons, not whichever one won the debate.

What each one actually does

RAG (retrieval-augmented generation) looks up relevant documents at the moment of the query and injects them into the prompt, so the model answers using material it’s never been trained on, with a citation trail back to the source. The model’s weights never change. New documents are usable the moment they’re indexed.

Fine-tuning adjusts the model’s weights on examples of the behavior you want, so the model does something differently by default, no retrieval step required. It’s not primarily a way to teach a model new facts, it’s a way to teach it a new shape: a tone, a format, a way of reasoning about your domain’s edge cases.

Long context is the least glamorous option and the most underrated: just put the relevant material directly in the prompt, every time, and let a long-context model handle it. No index to build, no weights to train, no infrastructure beyond a bigger prompt.

When RAG wins

Your source material changes often, is large (more than comfortably fits in a context window), and answers need to be traceable to a specific document. Customer support over a changing knowledge base, legal or compliance search over a filing archive, anything where “where did this answer come from” is a real requirement, not a nice-to-have. The query-time lookup means this morning’s document update is usable this afternoon, with no retraining cycle.

When fine-tuning wins

The problem isn’t “the model doesn’t know this fact,” it’s “the model’s default behavior is wrong for this domain.” A classifier that needs to apply your specific, non-obvious edge-case rules consistently. An agent that needs to sound like your brand’s support team, not like a generic assistant, on every single response. A narrow, high-volume task where fine-tuning a smaller model beats paying frontier-model prices for every call. Fine-tuning is a commitment: a new training run every time the desired behavior meaningfully shifts, so it fits best where the target behavior is genuinely stable.

When long context quietly wins

Smaller, curated datasets that fit comfortably in a modern context window, in internal tools where the cost of a slightly longer prompt isn’t the thing you’re optimizing, or in early-stage builds where you want to validate the product is worth building before investing in a retrieval pipeline. We default to long context more often than most teams expect, specifically because it’s the cheapest thing to get right and the easiest thing to rip out later if it stops scaling.

The combination that actually ships

Most production systems we build aren’t a single answer to this question. A support agent might use RAG for pulling the relevant policy documents, with a light fine-tune so its responses consistently match the company’s tone, and long context for the handful of most-common edge cases that are cheaper to just always include than to retrieve. Treating this as “pick one” is where teams waste the most time, usually by over-investing in a fine-tuning pipeline for a problem that a retrieval step would have solved in a week.

A decision checklist

Does the source material change weekly or more often? That’s a point toward RAG. Does the model need to behave differently by default, not just know more? That’s a point toward fine-tuning. Is the dataset small and the build timeline tight? Start with long context and add complexity only once you’ve proven you need it. We wrote more on that bias toward starting simple in why six-week projects beat six-month ones, and the same instinct applies here: the cheapest approach that could plausibly work is usually the one to try first.