Key takeaways

The through-line

The transformer won because it made sequence modelling parallel. Everything else — the block structure, the residual stream, the scaling laws, the post-training stack — follows from having removed the sequential bottleneck that made RNNs untrainable at scale.

The ideas that carry

• The residual stream is the architecture. Attention and the feed-forward network don't transform the representation so much as read from and write to a shared bus that runs the depth of the model. That framing explains residual connections, why normalisation placement matters, and why layers can be pruned or merged at all.
• Attention moves information between positions; the FFN does the computation. Roughly two-thirds of the parameters are in the FFN. It is where most of the knowledge lives.
• Pre-norm beat post-norm for a boring reason: trainability. Normalisation placement decides whether a 100-layer stack converges without warmup gymnastics.
• Decoder-only won on generality, not superiority. Encoder-decoder is still better where the input/output split is genuine; decoder-only became the default because one objective — next-token prediction — turned out to subsume nearly every task.
• Mixture of Experts decouples parameter count from compute. Route each token to a few experts and total parameters grow while per-token FLOPs don't. The hard part is load balancing and the memory to hold all those experts.
• Scaling laws made model design a budgeting exercise. Loss is predictable from compute, data and parameters — Chinchilla's correction was that most models were badly under-trained on data for their size.
• Pre-training gives capability; post-training gives usability. SFT, RLHF and DPO don't teach the model facts. They teach it which of the things it can already do it should actually do.

What you can now do

Read a model card and infer the architecture's trade-offs. Explain where parameters, compute and memory actually go. Justify a decoder-only vs. encoder-decoder choice, and a dense vs. MoE choice, against a real serving budget.

If one thing sticks

Almost every "new architecture" is a rearrangement of the same four pieces — attention, FFN, normalisation, residual. Knowing what each contributes lets you read any new model in an afternoon.