Pre-training — learning everything from next-token prediction

What a single training step actually looks like

Everything up to here described a model doing a forward pass. Training is that same forward pass, plus a comparison and a nudge. Shrink it to something you can see all of at once: a vocabulary of six words, and one position to predict.

The sentence is "the server rejected the request because it was ___", and the corpus says the next word is overloaded.

Before training, the model's output is close to flat — about 0.17 on each of the six words. This is not the model being stupid; it is the model being honest. Its weights are random, so it has no reason to prefer any word, and a flat distribution is the correct expression of knowing nothing.

The target is the opposite shape: 1.00 on the true word, 0.00 on everything else. Nothing in the corpus tells us the true word is 87% likely — the corpus contains one sentence, and in it, that word occurred.

The loss is the distance between those two columns, measured as cross-entropy, which for a one-hot target collapses to something simple: the negative log of the probability the model assigned to the correct word. At 0.17, that is 1.77. Every weight in the network then moves a little in the direction that would have made that number smaller.

After enough steps, the model puts 0.87 on overloaded and the loss is 0.14. Notice it did not go to 1.00 and should not. The remaining 0.13 is spread over slow, busy, down — words that genuinely could follow that clause in some other document. A model driven to 1.00 on every training token has memorised the corpus rather than learned the language.

Why this one trick teaches everything

The remarkable part is what falls out of an objective this narrow. Nobody supplies labels. The text is its own supervision — every position is simultaneously a training example, its answer being the token that happens to come next — which is why a trillion tokens of ordinary web text is a usable training set and why the causal mask from the attention course matters so much: it lets all those positions be trained in one forward pass without any of them peeking at its own answer.

And because "predict the next token" is asked of every kind of text, it quietly demands every kind of competence. Finishing a line of a proof requires the mathematics. Finishing a function requires the syntax and the intent. Finishing "the capital of Australia is ___" requires the fact. None of these were trained for; they are what minimising that one number over enough text forces into the weights.

The rest of this lesson covers how this is done at scale — the data pipeline, the optimiser, the hardware, and what the loss curve looks like when it works.

A transformer learns by predicting the next token. Given a sequence [t_1, t_2, ..., t_n], the model produces a probability distribution over the vocabulary at each position, and the training loss is the average negative log-probability of the actual next token across all positions:

This is the cross-entropy loss between the model's predicted distribution and the one-hot ground truth. A loss of 3.0 means the model assigns the correct next token an average probability of exp(-3.0) ≈ 0.05 — roughly 1 in 20. A loss of 1.5 corresponds to exp(-1.5) ≈ 0.22, or roughly 1 in 5. Frontier models (Llama 3 70B, GPT-4) achieve pre-training losses around 1.5–1.8 on held-out web text.

What next-token prediction learns

The cross-entropy objective imposes no explicit curriculum. The model receives no labels for "this is grammar," "this is factual knowledge," or "this is a reasoning step." Yet to minimize loss, the model must implicitly learn all of these:

• Syntax and grammar — predicting the next word in English requires knowing that "the" is likely followed by a noun or adjective, not a verb. These statistical regularities are captured in the model's attention patterns and FFN weights.
• World knowledge — "The capital of France is" has a strongly peaked next-token distribution: "Paris" dominates. To predict this correctly, the model must store factual associations.
• Reasoning patterns — in mathematical or logical text, the next token often follows deductively from prior context. The model learns approximate logical inference to minimize loss on these examples.
• Code — programming language syntax is highly structured. The model learns valid syntax, common library calls, algorithmic patterns, and even debugging patterns from code in the training corpus.
• Format following — JSON, XML, markdown, and structured data in the training set teach the model to produce syntactically valid structured output when the context implies a structured format.

The richness of capabilities emerges from the diversity of the training data, not from any architectural innovation beyond the basic transformer.

The data pipeline

Raw internet text is unusable for training. Converting a web crawl into a training-ready token stream requires a multi-stage pipeline that typically reduces the raw data by 5–10×.

Web crawl and extraction

Common Crawl (commoncrawl.org) is the standard starting point — a continuously updated crawl of the public web, currently totaling over 250 billion pages and several petabytes of compressed HTML. The pipeline begins with HTML extraction: stripping markup, navigation, boilerplate, and ads to recover the main content body. Libraries like trafilatura and resiliparse handle this, producing plain text from raw HTML.

Quality filtering

Not all extracted text is useful. Quality filtering removes:

• Duplicate and near-duplicate pages — The same content appears across many sites (scraped articles, syndicated content, boilerplate text). Exact deduplication removes byte-identical documents. Near-deduplication uses MinHash (Broder 1997) to estimate Jaccard similarity between documents and removes pairs exceeding a similarity threshold (typically 0.8). Llama 3's training pipeline used both URL-level and document-level deduplication.
• Low-quality text — Gibberish, machine-generated spam, keyword-stuffed SEO pages, and extremely short documents (fewer than ~50 words) are removed. Quality classifiers (often a small language model or a fastText classifier trained on curated examples) score each document, and those below a threshold are discarded. Llama 3 used a Llama 2-based quality classifier.
• Harmful and personally identifiable content — Safety classifiers flag hate speech, explicit content, and harassment. PII detection removes documents containing email addresses, phone numbers, and other personal identifiers. These filters are never perfect — they trade recall for precision, accepting some false negatives to avoid removing too much useful content.
• Line-level filtering — Beyond document-level filtering, individual lines with excessive punctuation, markup artifacts, or nonsensical character sequences are removed from otherwise good documents.

Deduplication at scale

Deduplication is computationally expensive. For Llama 3's ~100T raw tokens, the deduplication pipeline alone required thousands of CPU-hours. The standard approach:

• Exact substring deduplication — Using suffix arrays to find and remove repeated substrings longer than a threshold (typically 50–100 characters). This catches common boilerplate (cookie notices, privacy policies, navigation menus) that survived HTML extraction.
• MinHash deduplication — Each document is represented as a set of character n-grams (typically 5-grams). MinHash generates a compact signature (~128 hash values) for each document, and locality-sensitive hashing (LSH) groups documents with similar signatures. Documents within a group exceeding the similarity threshold are deduplicated (keeping one copy). This runs in approximately O(n) time versus the O(n^2) of comparing all pairs.
• Training-set deduplication against evaluation sets — To ensure honest benchmark evaluation, documents similar to evaluation benchmarks (MMLU, HellaSwag, etc.) are removed from training data. This is "contamination" filtering — Llama 3's technical report describes detecting and removing sequences with high n-gram overlap against common benchmarks.

Data composition

The filtered data is mixed from multiple sources in carefully tuned proportions. Llama 3's 15T token training set was approximately:

• ~50% web text (filtered Common Crawl and other web sources)
• ~25% code (GitHub, StackOverflow, code documentation)
• ~15% scientific and reference text (Wikipedia, arXiv, textbooks)
• ~10% multilingual text, books, and specialized corpora

These proportions significantly affect model behavior. Over-weighting code improves coding ability but can degrade conversational quality. Over-weighting formal text (Wikipedia, textbooks) improves factual accuracy but produces stilted, encyclopedic generations. The ratios are typically tuned empirically on a small model before committing to a full training run.

Tokenization and batching

After filtering, text is tokenized (using the model's trained tokenizer, typically BPE — via SentencePiece, tiktoken, or a custom implementation), shuffled at the document level to prevent the model from learning ordering artifacts, concatenated into long sequences with document separators, and packed into batches. Each training batch typically contains millions of tokens — Llama 3's batch size ramped from 4M to 16M tokens during training.

Training infrastructure

Training a frontier model requires coordinating thousands of GPUs for weeks. The engineering challenges are primarily about distributed computation and fault tolerance.

Llama 3 405B training setup

Meta's Llama 3 405B was trained on 16,384 NVIDIA H100 GPUs organized in a 3D parallelism scheme (the 70B model used a smaller cluster with the same parallelism strategy):

• Data parallelism (DP) — divide the batch across GPU groups. Each group processes a different subset of the batch through the full model, then synchronizes gradients via all-reduce. Llama 3 used DP degree of 128.
• Tensor parallelism (TP) — split individual weight matrices across GPUs within a single machine. For a linear layer with weight W of shape (d_model, d_ff), TP=8 gives each GPU a (d_model, d_ff/8) slice. The GPUs compute partial results and combine via all-reduce. TP=8 (one per GPU within a node) is standard because it requires high-bandwidth interconnect (NVLink, ~900 GB/s between H100s in the same node).
• Pipeline parallelism (PP) — split the model's layers across GPU groups. The 126 layers of Llama 3 405B are divided across multiple GPU groups, with micro-batches flowing through the pipeline. PP reduces memory per GPU (each group stores only its layers) at the cost of "pipeline bubbles" — idle time while GPUs wait for input from the previous stage. Llama 3 used PP degree of 16.

The total parallelism: $\text{DP} \times \text{TP} \times \text{PP} = 128 \times 8 \times 16 = 16, 384$ GPUs.

The training loop

import torch
import torch.nn as nn
from torch.optim import AdamW

def training_step(model, batch, optimizer, scaler, max_grad_norm=1.0):
    """
    Simplified training step (single GPU, no parallelism).
    Production training wraps this with FSDP/TP/PP.
    """
    input_ids = batch["input_ids"]        # (batch_size, seq_len)
    labels = batch["labels"]              # (batch_size, seq_len) — shifted by 1

    with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
        logits = model(input_ids)         # (batch_size, seq_len, vocab_size)

        # Cross-entropy loss: compare predicted distribution to actual next token
        loss = nn.functional.cross_entropy(
            logits.view(-1, logits.size(-1)),  # (batch_size * seq_len, vocab_size)
            labels.view(-1),                    # (batch_size * seq_len,)
            ignore_index=-100,                  # ignore padding tokens
        )

    # Backward pass with gradient scaling for mixed precision
    scaler.scale(loss).backward()

    # Gradient clipping — prevents exploding gradients from bad batches
    scaler.unscale_(optimizer)
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)

    # Optimizer step
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad(set_to_none=True)

    return loss.item()


def get_lr_schedule(step, warmup_steps=2000, max_steps=500000,
                    max_lr=3e-4, min_lr=3e-5):
    """
    Linear warmup + cosine decay schedule.
    """
    if step < warmup_steps:
        return max_lr * (step / warmup_steps)
    progress = (step - warmup_steps) / (max_steps - warmup_steps)
    import math
    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))


# Typical optimizer configuration for LLM pre-training
def create_optimizer(model, lr=3e-4, weight_decay=0.1):
    # Separate weight decay for different parameter groups
    decay_params = []
    no_decay_params = []
    for name, param in model.named_parameters():
        if param.requires_grad:
            if "bias" in name or "norm" in name or "embedding" in name:
                no_decay_params.append(param)
            else:
                decay_params.append(param)

    return AdamW([
        {"params": decay_params, "weight_decay": weight_decay},
        {"params": no_decay_params, "weight_decay": 0.0},
    ], lr=lr, betas=(0.9, 0.95), eps=1e-8)

Activation checkpointing

The backward pass needs the activations from the forward pass, and storing all of them is what actually caps trainable model size. For a 70B model at sequence length 8192, stored activations run to hundreds of gigabytes — more than the parameters, gradients and optimizer state combined.

Activation checkpointing (also called gradient checkpointing) trades compute for memory: keep only a few activations, at layer boundaries, and recompute the rest during the backward pass. Checkpointing every layer cuts activation memory from O(n_layers) to roughly O(sqrt(n_layers)) in the usual scheme, at the cost of one extra forward pass — about 30% more compute per step. Essentially every large pre-training run accepts that trade, because the alternative is not a slower run but an impossible one.

from torch.utils.checkpoint import checkpoint

class TransformerBlock(nn.Module):
    def forward(self, x, use_checkpoint=True):
        if use_checkpoint and self.training:
            # Recompute this block's activations during backward
            # instead of holding them for the whole forward pass.
            return checkpoint(self._forward, x, use_reentrant=False)
        return self._forward(x)

Mixed precision and loss scaling

The loop above wraps the forward pass in torch.autocast and the backward pass in a GradScaler. The scaler exists because of FP16's range, not its precision: FP16 underflows below about 6e-5, and small gradients — exactly the ones late in training — silently flush to zero. Loss scaling multiplies the loss by a large constant before backward(), pushing gradients up into representable range, then divides them out before the optimizer step. scaler.unscale_(optimizer) is what removes the factor, which is why it has to run before gradient clipping: clipping a scaled gradient would apply the wrong threshold.

BF16 needs none of this. It trades mantissa bits for exponent bits, matching FP32's dynamic range, so gradients never underflow and the scaler becomes a no-op. That is the reason large runs standardized on BF16 the moment the hardware supported it — one fewer thing that destabilizes a run you cannot afford to restart. Keep the scaler only for FP16 training on pre-Ampere hardware.

Learning rate schedule

Every frontier model uses the same schedule pattern: linear warmup from 0 to the peak learning rate over the first ~2000 steps, followed by cosine decay to a minimum learning rate (typically 1/10th of the peak). The peak learning rate scales inversely with model size — Llama 3 8B uses lr=3e-4, Llama 3 70B uses lr=1.5e-4. Larger models need smaller learning rates because each gradient step has a larger absolute effect on the loss landscape.

Gradient clipping

Gradient norms are clipped to a maximum value (typically 1.0) before each optimizer step. Without clipping, occasional batches containing unusual text (long sequences of repeated tokens, malformed data, or numerically unstable examples) can produce enormous gradients that destabilize training. Clipping rescales the gradient vector to have norm at most max_grad_norm without changing its direction.

Loss spikes

Pre-training loss curves are not smooth. Periodic sharp increases in loss — "spikes" — occur throughout training. These are typically caused by:

• Bad data batches — despite extensive filtering, some batches contain adversarial or degenerate text that produces unusually high loss and large gradients.
• Numerical instability — at bfloat16 precision, certain gradient accumulations can overflow or produce NaN values.
• Learning rate interactions — specific parameter configurations can create temporary instabilities, especially in the early phases of cosine decay.

Most spikes resolve spontaneously within 100–500 steps. If a spike persists or causes divergence, the standard recovery procedure is to roll back to a recent checkpoint and either skip the problematic batch or reduce the learning rate. Llama 3's training report describes encountering ~10 significant loss spikes during the 24-day training run, all of which recovered without intervention.

Checkpointing

Model state (parameters, optimizer state, learning rate schedule position, random number generator states) is saved to persistent storage every N steps. For Llama 3, checkpoints were saved every 1000 steps. Each checkpoint for a 70B model is approximately 500 GB (model weights in bfloat16 ~140 GB, optimizer states ~280 GB, gradients and metadata ~80 GB). Over a full training run, this produces petabytes of checkpoint data — typically managed with a rolling window that retains only the most recent K checkpoints plus a few earlier milestones.

Checkpointing also serves as the primary mechanism for handling GPU failures. At the scale of 16,384 GPUs running for 24 days, hardware failures are frequent — Meta's infrastructure reports describe GPU failures roughly every 2–3 hours at this scale. The training framework detects failures, restarts from the most recent checkpoint on replacement hardware, and continues. The overhead of re-computing lost steps (at most 1000 steps between checkpoints, typically 15–30 minutes of compute) is acceptable compared to losing the entire run.

The gap between pre-training and a useful model

A pre-trained model is not yet useful as an assistant. It generates text that continues the statistical patterns it learned — if you give it a question, it may generate five more questions (because questions often appear in lists on the web). Pre-training produces a powerful text completion engine. Turning it into a model that follows instructions, answers questions, and refuses harmful requests requires additional post-training stages: supervised fine-tuning (SFT) on instruction-response pairs, reinforcement learning from human feedback (RLHF), and other alignment techniques. These post-training stages use the same transformer weights and the same training loop mechanics (forward pass, loss, backward pass, optimizer step) — the difference is in the training data and loss function, not the infrastructure.