## The Decision Framework Prompt engineering modifies the input to a fixed model. Fine-tuning modifies the model weights themselves. RAG injects external documents into the context window at inference time. Each approach changes a different variable in the generation pipeline, and each has failure modes the others avoid. ### When fine-tuning wins Fine-tuning encodes behavior into parameters. The model learns to produce outputs that match a distribution — style, format, reasoning patterns — without needing explicit instructions at inference time. **Style and tone transfer.** A customer support model fine-tuned on 2,000 examples of brand-voice responses produces on-brand text without a 500-token system prompt. This eliminates the prompt tax: every token in the system prompt costs latency and money on every request. A Llama 3 8B fine-tuned for tone matches GPT-4's style adherence at 1/20th the per-token cost (measured on internal benchmarks at ~85% human preference parity). **Structured output compliance.** A model fine-tuned on 5,000 examples of `input → valid JSON` produces syntactically correct JSON 99.2% of the time vs 87% for the same base model with few-shot prompting (Zheng et al. 2024). The fine-tuned model internalizes the grammar rather than following instructions about it. **Domain specialization.** Medical, legal, and code models fine-tuned on domain corpora show 15-30% accuracy improvements on domain benchmarks. Med-PaLM 2 achieved 86.5% on MedQA through fine-tuning on medical reasoning chains (Singhal et al. 2023). A code model fine-tuned on internal APIs autocompletes proprietary function signatures that no general model has seen. **Latency reduction.** A Mistral 7B fine-tuned for summarization matches GPT-4's quality on your specific summarization task while running at 3x the throughput on a single A10G. Smaller fine-tuned models replace larger general models when the task is narrow enough. ```python # Cost comparison: API vs fine-tuned self-hosted # Assuming 1M requests/month, 500 tokens avg output # GPT-4o API api_cost_per_token = 10.0 / 1_000_000 # $10/M output tokens monthly_api_cost = 1_000_000 * 500 * api_cost_per_token # $5,000/month # Fine-tuned Llama 3 8B on A10G (24GB VRAM) gpu_hourly = 1.10 # A10G spot on AWS monthly_gpu = gpu_hourly * 730 # $803/month training_one_time = 50.0 # 4 hours on A100 for QLoRA # Total first month: $853, subsequent: $803/month # Breakeven vs API: month 1 ``` ### When RAG wins RAG wins when the knowledge base changes faster than you can retrain. A legal research tool indexing new case law daily cannot fine-tune on every update — the corpus shifts too fast. RAG retrieves the current document and grounds the answer in it. - Knowledge changes weekly or faster — RAG updates an index in minutes, fine-tuning takes hours - Citations are required — RAG can point to the exact chunk that sourced the answer - Access control matters — RAG filters retrievable documents per user; fine-tuned knowledge is baked into weights accessible to all users - The knowledge base exceeds what fits in training data — a 10M document corpus cannot be memorized into 8B parameters - Factual precision dominates — RAG with good retrieval produces fewer hallucinations on factual queries than fine-tuning alone (Lewis et al. 2020) ### When prompt engineering is enough Prompt engineering costs nothing to iterate. If you can solve the problem with a well-crafted system prompt and few-shot examples, skip fine-tuning entirely. - Task is general (translation, summarization, Q&A on provided text) - You need flexibility across many task variants in one deployment - Your dataset is under 100 high-quality examples (not enough for fine-tuning signal) - The frontier model already performs at 90%+ on your eval — fine-tuning a smaller model won't close the gap cheaply - You need rapid iteration — prompt changes deploy instantly, fine-tuning has a multi-hour feedback loop The prompt engineering ceiling is real, though. Once you exhaust few-shot examples (context window limits) and system prompt instructions (the model ignores long instructions past ~2000 tokens), you have hit the prompt ceiling. At that point, fine-tuning converts instruction-following into learned behavior. A concrete example: extracting structured data from medical notes. Prompt engineering with 5-shot examples achieves 78% field accuracy. Adding 10 shots improves to 82% but eats 3000 context tokens per request. Fine-tuning on 3000 examples reaches 94% with zero prompt overhead — the extraction logic lives in the weights. ### The decision tree ```python def choose_approach(task): if task.knowledge_changes_weekly: if task.needs_citations or task.needs_access_control: return "RAG" if task.needs_specific_style: return "RAG + fine-tuned generator" return "RAG" if task.base_model_accuracy > 0.90: if task.latency_budget_ms < 200: return "fine-tune smaller model" return "prompt engineering" if task.training_examples < 100: return "prompt engineering + synthetic data generation" if task.requires_format_compliance or task.requires_style_match: return "fine-tuning" if task.domain_specific and task.training_examples > 1000: return "fine-tuning" return "prompt engineering with eval, then fine-tune if ceiling hit" ``` ### Hybrid architectures Production systems rarely use one approach in isolation. The most effective pattern combines fine-tuning with RAG: fine-tune the model to follow a specific output format and reasoning style, then augment it with retrieved context for factual grounding. ```python # Hybrid: fine-tuned model + RAG # The fine-tuned model knows HOW to answer (format, style, reasoning) # RAG provides WHAT to answer about (current facts) from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained( "your-org/llama3-8b-support-finetuned", # fine-tuned for style device_map="auto", ) tokenizer = AutoTokenizer.from_pretrained("your-org/llama3-8b-support-finetuned") def generate_answer(query, retrieved_docs): context = "\n".join([doc.page_content for doc in retrieved_docs]) prompt = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|> You are a support agent. Use the provided context to answer accurately. Context: {context}<|eot_id|><|start_header_id|>user<|end_header_id|> {query}<|eot_id|><|start_header_id|>assistant<|end_header_id|> """ inputs = tokenizer(prompt, return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7) return tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True) ``` ### Cost modelling The real cost comparison must account for: training compute (one-time or periodic), inference compute (ongoing), iteration speed (how fast you can ship improvements), and failure cost (what happens when the model is wrong). - Full fine-tuning Llama 3 8B: ~$50-100 on A100 for 3 epochs over 10K examples - QLoRA fine-tuning same model: ~$10-20 on A10G, same data - Hosting inference (A10G, 8B model, vLLM): ~$800/month, handles ~50 req/s - GPT-4o API at 50 req/s sustained: ~$15,000/month - GPT-4o-mini API at 50 req/s: ~$2,000/month The crossover point: if your fine-tuned 8B model matches GPT-4o-mini quality on your specific task, self-hosting breaks even at ~40K requests/day. Below that volume, API calls win on operational simplicity. Fine-tuning is not a one-time cost. Distribution shift in your data means periodic retraining. Budget for monthly or quarterly refresh cycles. Track eval metrics in production — when accuracy drops below your threshold, trigger a retrain on fresh data. ### Common failure modes Fine-tuning fails when: - Training data is too small (< 100 examples): the model overfits to surface patterns and hallucinates in production. Minimum viable datasets: 500 examples for format compliance, 2000 for style transfer, 5000+ for domain knowledge. - Training data doesn't represent production distribution: a model fine-tuned on formal English support tickets fails on Slack-style informal queries. The training set must cover the full input distribution you expect in deployment. - The base model lacks the capability: fine-tuning cannot teach Llama 3 8B to do reliable multi-step mathematical reasoning that GPT-4 handles — the base model's architecture and pre-training limit the ceiling. Fine-tuning adapts existing capabilities to your task; it does not create new capabilities from scratch. - You evaluate on the wrong metric: a model with 0.9 training loss can still produce harmful outputs. Loss is necessary but not sufficient — always pair loss monitoring with task-specific evaluation (format validity rate, factual accuracy, human preference).