Nybble™
← Back to modules

LLM Training Pipeline — Advanced

Chinchilla vs Kaplan, compute vs inference optimality, AdamW internals, 3D parallelism, ZeRO stages, stability/loss spikes, MFU, reward modeling, PPO vs DPO, RLAIF, reward hacking.

Hard50 questions

Sample questions

1

Chinchilla revised the earlier Kaplan scaling laws mainly by changing which conclusion?

  • Loss no longer follows any power law with scale
  • Larger models are always worse than smaller ones
  • For a fixed compute budget, data should scale with model size
  • Compute budget is entirely irrelevant to the final loss reached here

Why

Kaplan's analysis favored spending compute mostly on model size; Chinchilla corrected this, showing tokens and parameters should scale together, so prior models were undertrained. Power-law scaling still holds, bigger isn't always worse, and compute clearly bounds the achievable loss.

Training Compute-Optimal LLMs (Chinchilla)Scaling Laws for Neural Language Models
2

LLaMA deliberately trained relatively small models on far more tokens than Chinchilla-optimal. What is the rationale?

  • It minimizes the total pretraining compute spent
  • A smaller model is cheaper to serve at inference
  • It makes the reward model easier to train later
  • It is required for mixed-precision training to work

Why

Chinchilla optimizes training compute, but LLaMA optimizes the whole lifecycle: overtraining a smaller model costs more to train yet is much cheaper to serve for the model's lifetime. It doesn't minimize training compute, ease reward modeling, or gate mixed precision.

LLaMA: Open and Efficient Foundation LMsTraining Compute-Optimal LLMs (Chinchilla)
3

You have a fixed inference budget and expect to serve billions of tokens. How should that change your training choice versus pure Chinchilla-optimal?

  • Train a larger model on fewer tokens to save time
  • Skip alignment to reduce total pipeline cost
  • Train a smaller model on extra tokens beyond optimal
  • Use FP32 everywhere for maximum accuracy

Why

When serving dominates cost, you 'overtrain' a smaller model past the compute-optimal token count, accepting higher training cost for far cheaper inference. A larger model raises serving cost, skipping alignment hurts quality, and FP32 wastes memory/throughput.

LLaMA: Open and Efficient Foundation LMsTraining Compute-Optimal LLMs (Chinchilla)
4

In AdamW, how does the weight-decay term differ from L2 regularization folded into Adam's gradient?

  • Decay is applied to weights, decoupled from the adaptive step
  • Decay is scaled by the adaptive second-moment estimate each step
  • Decay is added to the loss before backpropagation
  • Decay only affects the bias terms, not the weights

Why

AdamW decouples weight decay so it directly shrinks the weights, independent of the adaptive per-parameter scaling; classic 'Adam + L2' instead routes the penalty through the moment estimates, weakening it for large-gradient parameters. It isn't added to the loss here, nor limited to biases.

Decoupled Weight Decay Regularization (AdamW)Scaling Laws for Neural Language Models
5

For a model with P parameters trained in mixed precision with Adam, why do optimizer states dominate memory?

  • Adam recomputes all activations at every step
  • Adam keeps a full copy of the training dataset
  • Adam duplicates the attention cache for each layer
  • Adam stores two FP32 moment tensors per parameter

Why

Adam holds first- and second-moment estimates (typically FP32), plus an FP32 master copy, so per-parameter optimizer memory can exceed the 16-bit weights several-fold — the motivation for ZeRO sharding. It doesn't store activations, the dataset, or attention caches as optimizer state.

ZeRO: Memory Optimizations Toward Trillion-Param ModelsDecoupled Weight Decay Regularization (AdamW)
Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

  • Every question in this module
  • Instant feedback and supporting reading
  • Your score and progress, saved

Free · your email is used for progress only.

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack