RAG vs. fine-tuning vs. long context vs. tools

Four levers, four different jobs

"The model can't do X" has four common fixes, and they are not interchangeable. Picking the wrong one is one of the most expensive mistakes in applied LLM work, because it sends you building the wrong system for weeks. The trick is to notice that each lever changes a different thing.

• RAG changes what the model knows, at query time.
• Fine-tuning changes how the model behaves.
• Long context changes how much you can show it at once.
• Tools / agents change what the model can do.

Match the lever to the kind of gap you actually have.

RAG — when the gap is knowledge

Reach for RAG when the model's problem is that it doesn't know something: facts that are private, fresh, voluminous, or that change often. RAG shines precisely because the knowledge lives outside the weights — you update it by editing an index, and you get attribution for free.

Signs it's a RAG problem:

• The answer depends on documents the model never trained on (your wiki, contracts, tickets).
• The facts change faster than you could ever retrain.
• Users need to see where an answer came from.
• The knowledge base is large — far more than you would want to, or could, stuff into every prompt.

Fine-tuning — when the gap is behavior

Fine-tuning continues training on your examples to shift the model's default behavior: its tone, its format, its adherence to a schema, its skill at a narrow, repetitive task. It is about how the model responds, not what it knows.

Signs it's a fine-tuning problem:

• You need a consistent voice, style, or strict output format that prompting alone keeps drifting from.
• You have a narrow, high-volume task where a smaller fine-tuned model could match a big model at lower cost and latency.
• You have a solid set of input→output examples that demonstrate the behavior you want.

The classic mistake is trying to fine-tune in knowledge. It can appear to work on a tiny set and then fail: the model overfits, forgets other things, or reproduces facts unreliably and unattributable. As a rule of thumb: fine-tune to change behavior, retrieve to change knowledge. They compose well — a fine-tuned model can be the generator inside a RAG system.

Long context — when you can just show it everything

Modern models accept very large inputs, which invites a fair question: if you can paste an entire document — or ten — into the prompt, why retrieve at all? Sometimes you shouldn't. If the whole relevant corpus reliably fits in the context window and is small and stable, skip the machinery and just include it.

But long context is not a general replacement for retrieval, for four reasons:

• Cost and latency. You pay for every token on every call. Stuffing 100K tokens of context into each query, when 2K would do, is wasteful at scale and slower to respond.
• It doesn't scale with the corpus. A knowledge base of millions of documents cannot fit in any window. Retrieval's whole job is to select the relevant slice, and that need never goes away.
• "Lost in the middle." Models attend unevenly across a long context, systematically underusing material buried in the middle (Liu et al., 2023). More context is not linearly more useful; a few precisely-retrieved passages often beat a giant dump.
• No selection, no attribution. A wall of context tells you nothing about which part mattered. Retrieval is the selection step that makes grounding and citation possible.

Long context and RAG are complements: bigger windows make RAG more forgiving (you can afford more chunks, larger ones), but you still retrieve to decide what goes in the window.

Tools and agents — when the gap is action

Sometimes the model doesn't need to know more or sound different — it needs to do something: run a calculation, query a live database, call an API, book the meeting. That is a job for tools, where the model emits a structured call and your runtime executes it. (Nybble has a whole "Build Your First AI Agent" track on this.)

Note that RAG is, in a sense, the simplest tool: "search a knowledge base." Agentic RAG — which we reach in the advanced section — blurs the line deliberately, letting the model decide when and what to retrieve rather than always retrieving once up front.

They combine — real systems use several

The decision tree above finds your primary lever, but production systems rarely use just one. A mature support assistant might: retrieve policy and account docs (RAG), on top of a base model fine-tuned to the company's voice and refusal style, with tools to look up live order status, all inside a generous context window that lets it hold several retrieved passages at once. The skill is not picking one lever forever — it's knowing which gap each part of the problem is, and reaching for the lever built for that gap.

A quick decision checklist

• New/changing/private facts, and you want citations → RAG.
• Consistent behavior, tone, or format, or a cheap specialist model → fine-tune.
• The whole relevant corpus is small and stable → consider long context alone.
• The model needs to act on live systems → tools / agents.
• Most real problems are a blend — decompose, then apply the right lever per part.

Key takeaways

• The four levers fix four different gaps: RAG = knowledge, fine-tuning = behavior, long context = how much you can show at once, tools = actions.
• Don't fine-tune to add facts; don't assume a big context window retires retrieval — cost, scale, "lost in the middle," and attribution all still favor selecting the right passages.
• Identify the kind of gap first, then pick the lever built for it — and expect real systems to combine several.