LLM Training Pipeline — Core

Pretraining objective, data curation, scaling laws, Chinchilla, AdamW, LR schedules, mixed precision, distributed training, SFT, RLHF, DPO, training evaluation.

Medium50 questions

Sample questions

What is the standard pretraining objective for a GPT-style (decoder-only) large language model?

Predicting the next token given all previous tokens
Predicting randomly masked tokens using both directions
Classifying whether two sentences are consecutive or not
Reconstructing the input from a compressed latent vector

Why

Decoder-only LMs use causal (autoregressive) language modeling — predict token t from tokens 1..t-1 by minimizing cross-entropy. Masked-token prediction is BERT's objective, next-sentence classification is an auxiliary BERT task, and latent reconstruction describes autoencoders.

Language Models are Few-Shot Learners (GPT-3)Scaling Laws for Neural Language Models

Your pretraining dashboard shows perplexity steadily dropping over the run. What does that falling number tell you?

The model uses fewer parameters to store knowledge
The optimizer is running at a higher learning rate
The model assigns higher probability to the held-out text
The training data contains fewer duplicate documents

Why

Perplexity is the exponential of the average cross-entropy, so lower perplexity means the model predicts the next tokens with higher probability — a better language model. It isn't a measure of parameter count, learning rate, or data duplication.

Scaling Laws for Neural Language Models Language Models are Few-Shot Learners (GPT-3)

What did the Chinchilla paper conclude about compute-optimal training?

Model size should grow while data is held constant
Data should grow while the model size is held constant
Model size and training tokens should scale together
Only the batch size needs to scale with compute budget

Why

Chinchilla showed that for a fixed compute budget, parameters and training tokens should scale roughly equally, so many earlier models were undertrained. Holding either factor fixed, or scaling only batch size, is exactly the imbalance Chinchilla corrected.

Training Compute-Optimal LLMs (Chinchilla)Scaling Laws for Neural Language Models

You have a fixed GPU-hours budget to pretrain a model and want the best possible loss. What does the Chinchilla result tell you to avoid?

Training a huge model on too few tokens
Using a cosine learning-rate decay schedule
Deduplicating the pretraining corpus first
Using the AdamW optimizer for the run

Why

Chinchilla shows a giant model trained on too little data is compute-inefficient; a smaller model trained on proportionally more tokens reaches lower loss for the same budget. Cosine decay, deduplication, and AdamW are all standard good practices, not things to avoid.

Training Compute-Optimal LLMs (Chinchilla)LLaMA: Open and Efficient Foundation LMs

Scaling laws for language models (Kaplan et al.) describe the test loss as which function of model size, data, and compute?

A step function with sudden drops at thresholds
A smooth power-law relationship over many orders
A linear decrease down to exactly zero loss
An oscillating curve that depends on random seeds

Why

Kaplan et al. found loss falls as a power law in parameters, dataset size, and compute across many orders of magnitude — smooth and predictable, not a step function or a line to zero, and not seed-dependent oscillation.

Scaling Laws for Neural Language Models Training Compute-Optimal LLMs (Chinchilla)

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack