Scaling laws — predicting loss from compute, data, and parameters
Cross-entropy loss on held-out text follows power-law relationships with three variables: the number of model parameters N, the dataset size D (in tokens), and the training compute budget C (in FLOPs). These relationships, first characterized by Kaplan et al. (2020, OpenAI), allow researchers to predict the performance of large models from the performance of small ones, turning model development from guesswork into (approximate) engineering.
The Kaplan scaling laws
Kaplan et al. trained over 400 transformer language models ranging from 768 parameters to 1.5 billion parameters and fit power-law curves to the relationship between each variable and held-out loss:
L(N) ≈ (N_c / N)^alpha_N where alpha_N ≈ 0.076 and N_c ≈ 8.8 × 10^13
L(D) ≈ (D_c / D)^alpha_D where alpha_D ≈ 0.095 and D_c ≈ 5.4 × 10^13
L(C) ≈ (C_c / C)^alpha_C where alpha_C ≈ 0.050 and C_c ≈ 3.1 × 10^8
Each law holds when the other two variables are not the bottleneck. L(N) describes loss when the model is trained on effectively infinite data with sufficient compute — the "parameter-limited" regime. L(D) describes loss when the model is large enough that data is the constraint. L(C) describes loss under a fixed total compute budget, optimized over both N and D.
The exponents are small (0.05–0.10), meaning returns diminish rapidly. Doubling compute reduces loss by a factor of 2^0.05 ≈ 1.035 — about 3.5%. To halve the loss, you need roughly 2^(1/0.05) ≈ 2^20 ≈ 10^6 times more compute. This is why frontier model training costs grow exponentially with each generation.
The compute-optimal question
Given a fixed compute budget C, how should you split it between model size N and training tokens D? Compute is approximately C ≈ 6 * N * D FLOPs for a dense transformer (6 FLOPs per parameter per token: 2 for the forward pass, 4 for the backward pass, accounting for the multiply-accumulate structure).
Kaplan et al. concluded that model size should scale faster than dataset size. Their recommendation: allocate most of the compute budget to a larger model trained on comparatively less data. This led to an era of "big model, small data" training — GPT-3 (175B parameters) trained on 300B tokens, a ratio of ~1.7 tokens per parameter.
Chinchilla — a different optimal
Hoffmann et al. (2022, DeepMind) ran a more systematic experiment: train over 400 models from 70M to 16B parameters, varying both model size and token count, and directly measure compute-optimal configurations. Their conclusion sharply contradicted Kaplan:
The compute-optimal ratio is approximately 20 tokens per parameter.
A 7B model should optimally train on ~140B tokens. A 70B model should train on ~1.4T tokens. The Chinchilla model itself — 70B parameters trained on 1.4T tokens — outperformed DeepMind's own Gopher (280B parameters trained on 300B tokens) despite using the same total compute budget.
The implication was dramatic: most existing large models were severely undertrained. GPT-3's 1.7 tokens per parameter was ~12× below the Chinchilla optimal. BLOOM (176B params, 341B tokens) was ~10× undertrained. PaLM (540B params, 780B tokens) was ~14× undertrained.
The Chinchilla scaling law for compute-optimal training:
N_opt ∝ C^0.50 and D_opt ∝ C^0.50
Both model size and data size should grow at the same rate with compute — a 10× increase in compute budget should be split equally: ~3.2× larger model trained on ~3.2× more tokens.
The Llama departure — inference-aware scaling
Chinchilla answers the question "what minimizes training loss for a given training compute budget?" But in production, training is a one-time cost and inference runs continuously. The question that matters for deployment is: "what maximizes quality per inference FLOP?"
The Llama family (Touvron et al. 2023) deliberately over-trains smaller models far past the Chinchilla optimum:
- Llama 1 7B — trained on 1T tokens (Chinchilla optimal: ~140B). Over-trained by ~7×.
- Llama 2 7B — trained on 2T tokens. Over-trained by ~14×.
- Llama 3 8B — trained on 15T tokens. Over-trained by ~94×.
Why this works: each additional token of training improves the model's quality, even past the Chinchilla point — just at diminishing returns for the training compute spent. But the trained model's quality is permanent. A Llama 3 8B trained on 15T tokens is substantially better than a Chinchilla-optimal 8B trained on 160B tokens, and both cost exactly the same to run at inference. The extra training cost is amortized across potentially billions of inference requests.
The practical scaling law for engineers becomes: for a fixed inference budget (maximum model size that fits your serving hardware), train on as much data as you can afford. The "optimal" training compute is determined by how much you're willing to spend on a one-time training run, not by any Chinchilla-like compute split.
Compute estimates for training runs
The approximation C ≈ 6 * N * D lets you estimate training costs:
def estimate_training_cost(
num_params_billions: float,
num_tokens_trillions: float,
gpu_type: str = "H100",
):
N = num_params_billions * 1e9
D = num_tokens_trillions * 1e12
C = 6 * N * D # total FLOPs
gpu_specs = {
"H100": {"bf16_tflops": 990, "cost_per_hour": 3.50},
"A100": {"bf16_tflops": 312, "cost_per_hour": 2.00},
"H800": {"bf16_tflops": 990, "cost_per_hour": 2.80},
}
spec = gpu_specs[gpu_type]
peak_flops = spec["bf16_tflops"] * 1e12
mfu = 0.40 # typical model FLOPs utilization
effective_flops = peak_flops * mfu
gpu_seconds = C / effective_flops
gpu_hours = gpu_seconds / 3600
# With N GPUs, wall-clock time scales roughly as gpu_hours / N
# (ignoring communication overhead)
cost = gpu_hours * spec["cost_per_hour"]
print(f"Model: {num_params_billions}B params, {num_tokens_trillions}T tokens")
print(f"Total compute: {C:.2e} FLOPs")
print(f"GPU: {gpu_type} ({spec['bf16_tflops']} BF16 TFLOPS, {mfu:.0%} MFU)")
print(f"GPU-hours: {gpu_hours:,.0f}")
print(f"Estimated cost: ${cost:,.0f}")
print(f"With 1024 GPUs: {gpu_hours / 1024:,.0f} hours ({gpu_hours / 1024 / 24:,.1f} days)")
print()
estimate_training_cost(8, 15, "H100") # Llama 3 8B
estimate_training_cost(70, 15, "H100") # Llama 3 70B
estimate_training_cost(671, 14.8, "H800") # DeepSeek-V3 (MoE, but ~37B active)Running this with the Llama 3 70B parameters: FLOPs. At 40% MFU on H100s, that's ~16M GPU-hours — broadly consistent with Meta's reported 24 days on 16,384 H100 GPUs (16,384 × 24 × 24 ≈ 9.4M GPU-hours, implying higher effective MFU with their optimized training stack).
Emergent abilities and the limits of prediction
Scaling laws predict aggregate loss (perplexity) with remarkable precision — but specific capabilities don't track loss smoothly. Wei et al. (2022, Google) documented "emergent abilities": capabilities that appear suddenly at specific scales rather than improving gradually.
Examples:
- Multi-step arithmetic — Models below ~10B parameters score near-zero on 3-digit multiplication. At ~50B parameters, accuracy jumps from near-zero to ~40%. At 175B+, it reaches ~80%.
- In-context learning — The ability to learn new patterns from examples in the prompt improves gradually with scale, but certain ICL tasks (translation from prompts, instruction following) appear discontinuously.
- Chain-of-thought reasoning — Prompting a model with "Let's think step by step" has no effect below ~60B parameters. Above that threshold, it dramatically improves accuracy on reasoning benchmarks (GSM8K, StrategyQA).
The mechanism behind emergent abilities is debated. Schaeffer et al. (2023) argued that emergence is partly an artifact of metric choice — when you measure accuracy (binary: right or wrong), performance looks discontinuous; when you measure token-level log-likelihood (continuous), improvement is gradual. Others argue that genuine phase transitions occur as model representations cross thresholds of compositional capability.
For practitioners, the implication is clear: scaling laws tell you how much compute you need for a given perplexity target, but they cannot reliably predict when a specific reasoning capability will appear. You have to train the model and test it.
Scaling laws for downstream tasks
Scaling laws extend beyond pre-training loss. Isik et al. (2024) and others have characterized power-law relationships for downstream task performance:
accuracy(C) ≈ a - b * C^(-gamma)
where a is the asymptotic accuracy (often near 1.0), b is a task-dependent offset, and gamma is the scaling exponent. The exponent gamma varies by task: simple classification tasks have gamma ≈ 0.1–0.2 (saturate quickly), while complex reasoning tasks have gamma ≈ 0.02–0.05 (improve slowly, need enormous compute).
This means the gap between "easy" and "hard" tasks grows with scale. A 7B model might score 80% on sentiment classification and 30% on mathematical reasoning. A 70B model might score 90% on sentiment (near saturation) and 55% on math (still climbing). The last 10% of accuracy on hard tasks costs orders of magnitude more compute than the first 80%.