Nybble™
LLM Fine-Tuning for Production

Lesson 1 of 14

The Decision Framework

Prompt engineering modifies the input to a fixed model. Fine-tuning modifies the model weights themselves. RAG injects external documents into the context window at inference time. Each approach changes a different variable in the generation pipeline, and each has failure modes the others avoid.

When fine-tuning wins

Fine-tuning encodes behavior into parameters. The model learns to produce outputs that match a distribution — style, format, reasoning patterns — without needing explicit instructions at inference time.

Style and tone transfer. A customer support model fine-tuned on 2,000 examples of brand-voice responses produces on-brand text without a 500-token system prompt. This eliminates the prompt tax: every token in the system prompt costs latency and money on every request. A Llama 3 8B fine-tuned for tone matches GPT-4's style adherence at 1/20th the per-token cost (measured on internal benchmarks at ~85% human preference parity).

Structured output compliance. A model fine-tuned on 5,000 examples of input → valid JSON produces syntactically correct JSON 99.2% of the time vs 87% for the same base model with few-shot prompting (Zheng et al. 2024). The fine-tuned model internalizes the grammar rather than following instructions about it.

Domain specialization. Medical, legal, and code models fine-tuned on domain corpora show 15-30% accuracy improvements on domain benchmarks. Med-PaLM 2 achieved 86.5% on MedQA through fine-tuning on medical reasoning chains (Singhal et al. 2023). A code model fine-tuned on internal APIs autocompletes proprietary function signatures that no general model has seen.

Latency reduction. A Mistral 7B fine-tuned for summarization matches GPT-4's quality on your specific summarization task while running at 3x the throughput on a single A10G. Smaller fine-tuned models replace larger general models when the task is narrow enough.

# Cost comparison: API vs fine-tuned self-hosted
# Assuming 1M requests/month, 500 tokens avg output

# GPT-4o API
api_cost_per_token = 10.0 / 1_000_000  # $10/M output tokens
monthly_api_cost = 1_000_000 * 500 * api_cost_per_token  # $5,000/month

# Fine-tuned Llama 3 8B on A10G (24GB VRAM)
gpu_hourly = 1.10  # A10G spot on AWS
monthly_gpu = gpu_hourly * 730  # $803/month
training_one_time = 50.0  # 4 hours on A100 for QLoRA
# Total first month: $853, subsequent: $803/month
# Breakeven vs API: month 1

When RAG wins

RAG wins when the knowledge base changes faster than you can retrain. A legal research tool indexing new case law daily cannot fine-tune on every update — the corpus shifts too fast. RAG retrieves the current document and grounds the answer in it.

  • Knowledge changes weekly or faster — RAG updates an index in minutes, fine-tuning takes hours
  • Citations are required — RAG can point to the exact chunk that sourced the answer
  • Access control matters — RAG filters retrievable documents per user; fine-tuned knowledge is baked into weights accessible to all users
  • The knowledge base exceeds what fits in training data — a 10M document corpus cannot be memorized into 8B parameters
  • Factual precision dominates — RAG with good retrieval produces fewer hallucinations on factual queries than fine-tuning alone (Lewis et al. 2020)

When prompt engineering is enough

Prompt engineering costs nothing to iterate. If you can solve the problem with a well-crafted system prompt and few-shot examples, skip fine-tuning entirely.

  • Task is general (translation, summarization, Q&A on provided text)
  • You need flexibility across many task variants in one deployment
  • Your dataset is under 100 high-quality examples (not enough for fine-tuning signal)
  • The frontier model already performs at 90%+ on your eval — fine-tuning a smaller model won't close the gap cheaply
  • You need rapid iteration — prompt changes deploy instantly, fine-tuning has a multi-hour feedback loop

The prompt engineering ceiling is real, though. Once you exhaust few-shot examples (context window limits) and system prompt instructions (the model ignores long instructions past ~2000 tokens), you have hit the prompt ceiling. At that point, fine-tuning converts instruction-following into learned behavior.

A concrete example: extracting structured data from medical notes. Prompt engineering with 5-shot examples achieves 78% field accuracy. Adding 10 shots improves to 82% but eats 3000 context tokens per request. Fine-tuning on 3000 examples reaches 94% with zero prompt overhead — the extraction logic lives in the weights.

The decision tree

def choose_approach(task):
    if task.knowledge_changes_weekly:
        if task.needs_citations or task.needs_access_control:
            return "RAG"
        if task.needs_specific_style:
            return "RAG + fine-tuned generator"
        return "RAG"

    if task.base_model_accuracy > 0.90:
        if task.latency_budget_ms < 200:
            return "fine-tune smaller model"
        return "prompt engineering"

    if task.training_examples < 100:
        return "prompt engineering + synthetic data generation"

    if task.requires_format_compliance or task.requires_style_match:
        return "fine-tuning"

    if task.domain_specific and task.training_examples > 1000:
        return "fine-tuning"

    return "prompt engineering with eval, then fine-tune if ceiling hit"

Hybrid architectures

Production systems rarely use one approach in isolation. The most effective pattern combines fine-tuning with RAG: fine-tune the model to follow a specific output format and reasoning style, then augment it with retrieved context for factual grounding.

# Hybrid: fine-tuned model + RAG
# The fine-tuned model knows HOW to answer (format, style, reasoning)
# RAG provides WHAT to answer about (current facts)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "your-org/llama3-8b-support-finetuned",  # fine-tuned for style
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("your-org/llama3-8b-support-finetuned")

def generate_answer(query, retrieved_docs):
    context = "\n".join([doc.page_content for doc in retrieved_docs])
    prompt = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a support agent. Use the provided context to answer accurately.

Context:
{context}<|eot_id|><|start_header_id|>user<|end_header_id|>
{query}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
    return tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)

Cost modelling

The real cost comparison must account for: training compute (one-time or periodic), inference compute (ongoing), iteration speed (how fast you can ship improvements), and failure cost (what happens when the model is wrong).

  • Full fine-tuning Llama 3 8B: ~$50-100 on A100 for 3 epochs over 10K examples
  • QLoRA fine-tuning same model: ~$10-20 on A10G, same data
  • Hosting inference (A10G, 8B model, vLLM): ~$800/month, handles ~50 req/s
  • GPT-4o API at 50 req/s sustained: ~$15,000/month
  • GPT-4o-mini API at 50 req/s: ~$2,000/month

The crossover point: if your fine-tuned 8B model matches GPT-4o-mini quality on your specific task, self-hosting breaks even at ~40K requests/day. Below that volume, API calls win on operational simplicity.

Fine-tuning is not a one-time cost. Distribution shift in your data means periodic retraining. Budget for monthly or quarterly refresh cycles. Track eval metrics in production — when accuracy drops below your threshold, trigger a retrain on fresh data.

Common failure modes

Fine-tuning fails when:

  • Training data is too small (< 100 examples): the model overfits to surface patterns and hallucinates in production. Minimum viable datasets: 500 examples for format compliance, 2000 for style transfer, 5000+ for domain knowledge.
  • Training data doesn't represent production distribution: a model fine-tuned on formal English support tickets fails on Slack-style informal queries. The training set must cover the full input distribution you expect in deployment.
  • The base model lacks the capability: fine-tuning cannot teach Llama 3 8B to do reliable multi-step mathematical reasoning that GPT-4 handles — the base model's architecture and pre-training limit the ceiling. Fine-tuning adapts existing capabilities to your task; it does not create new capabilities from scratch.
  • You evaluate on the wrong metric: a model with 0.9 training loss can still produce harmful outputs. Loss is necessary but not sufficient — always pair loss monitoring with task-specific evaluation (format validity rate, factual accuracy, human preference).