Modern architecture choices — the 2026 consensus

The transformer architecture as deployed in 2026 production models has converged on a remarkably consistent set of component choices. This convergence happened through empirical competition: hundreds of ablation studies across Llama, Mistral, Qwen, DeepSeek, and other model families tested alternatives for every component, and the same winners kept emerging.

What has converged

Nearly every production model released since late 2023 uses the same set of core components:

Grouped-query attention (GQA)

Standard multi-head attention gives every head its own key and value projections. GQA shares KV projections across groups of query heads — typically 8 KV heads serve 32 or 64 query heads. This reduces the KV cache size proportionally (a 4:1 grouping cuts KV cache by 4×) with negligible quality loss. Ainslie et al. 2023 (Google) showed that GQA with 8 groups matches full multi-head attention quality on language modeling while enabling 4–8× longer sequences at the same memory budget.

Every model in the current generation uses GQA: Llama 3 (8 KV heads for 32 or 64 query heads), Mistral 7B (8 KV heads), Qwen2 (8 KV heads), DeepSeek-V3 (uses multi-head latent attention, a further compression of the KV cache concept).

Rotary positional embeddings (RoPE)

RoPE (Su et al. 2021) encodes position by rotating the query and key vectors in 2D subspaces. The rotation angle for position m and dimension pair (2i, 2i+1) is m * theta_i, where $$\text{theta}_{i} = 10000^{ - 2\frac{i}{d_{\text{head}}}}$$. This produces a relative position encoding that decays smoothly with distance and extends naturally to longer sequences than seen during training (via frequency scaling techniques like YaRN, NTK-aware scaling, or ABF).

RoPE has replaced all other positional encoding schemes. Learned absolute embeddings (GPT-2, BERT), sinusoidal embeddings (original transformer), and ALiBi (BLOOM) are not used in any major 2026 model.

SwiGLU activation

The feed-forward network in the original transformer used ReLU: $\text{FFN}(x) = \max(0, \text{xW}_1 + b_1)W_2 + b_2$. Modern models use SwiGLU (Shazeer 2020):

SwiGLU introduces a gating mechanism — the SiLU(xW_gate) term controls which features pass through, while xW_up provides the values. This adds a third weight matrix (W_gate) to the FFN, increasing parameter count by ~50% per FFN block. But the quality improvement is consistent across scales: Shazeer's ablations show SwiGLU outperforming ReLU, GELU, and other variants at equal compute budget.

The intermediate dimension is typically 8/3 × d_model rounded to a multiple of 256 (to align with GPU tensor core sizes). For $d_{\text{model}} = 4096$, this gives $d_{\text{ff}} = 14336$ (Llama 3 8B, Mistral 7B). For $d_{\text{model}} = 8192$, $d_{\text{ff}} = 28672$ (Llama 3 70B).

RMSNorm

Layer normalization (Ba et al. 2016) normalizes activations by subtracting the mean and dividing by the standard deviation, then applies a learned scale and shift. RMSNorm (Zhang & Sennrich 2019) drops the mean subtraction and shift, normalizing only by the root mean square:

This is simpler, faster (no mean computation, no bias parameter), and empirically equivalent to LayerNorm for transformer language models. RMSNorm is universal in 2026 models.

Pre-norm placement

The original transformer ("post-norm") applied normalization after the residual addition: x = LayerNorm(x + Sublayer(x)). Pre-norm applies normalization before the sublayer: x = x + Sublayer(Norm(x)). Pre-norm training is more stable — gradients flow through the residual connection without passing through the normalization, preventing vanishing gradients in deep networks. All models from GPT-2 onward use pre-norm.

Weight tying

The embedding matrix (which converts token IDs to vectors at the input) and the language modeling head (which converts final hidden states back to token logits) share the same weight matrix, transposed. For a vocabulary of 128K tokens and $d_{\text{model}} = 4096$, this saves $128, 000 \times 4096 \times 2 \text{bytes} = ~1\text{GB}$ of parameters in float16 — significant for smaller models. Gemma and Qwen2 tie weights. Llama 3 (both 8B and 70B), Mistral 7B, and DeepSeek-V3 use untied embeddings, giving the output head more flexibility at the cost of additional parameters.

What diverges

MoE vs. dense

The largest architectural split in 2026 is whether to use Mixture of Experts. Mixtral 8x7B, DeepSeek-V3, Grok-1, and (reportedly) GPT-4 use MoE. Llama 3, Claude, Gemma, and Qwen2 are dense. The choice depends on the deployment scenario: MoE trades memory for compute efficiency, which favors high-throughput serving; dense models have simpler deployment and more predictable per-token latency.

Context length strategy

Models achieve long context through different mechanisms:

• RoPE frequency scaling — Llama 3 uses adjusted base frequency (theta=500,000 instead of the original 10,000) to extend the effective context. Combined with training on 128K-token sequences.
• Sliding window attention — Mistral 7B restricts each layer's attention to a local window (W=4096 tokens), with information propagating across windows through layers. This gives O(n × W) attention cost instead of O(n^2) but limits single-hop long-range dependencies.
• Brute-force training on long sequences — Most 2026 models simply train on longer sequences with standard full attention, relying on hardware improvements (FlashAttention, ring attention) to manage the O(n^2) cost.

Vocabulary size

Vocabulary sizes vary from 32K to 200K+ tokens:

• Llama 3 — 128,256 tokens (expanded from Llama 2's 32K to improve multilingual and code compression)
• Mistral 7B — 32,000 tokens
• Qwen2 — 152,064 tokens
• GPT-4o — ~200,019 tokens (o200k_base encoding)
• Gemma 2 — 256,000 tokens

Larger vocabularies compress text more efficiently (fewer tokens per sentence = faster inference and longer effective context) but increase the embedding matrix size and make the softmax over the vocabulary more expensive. The trend is clearly upward — no model released in 2025 or 2026 uses fewer than 32K tokens, and most use 100K+.

Concrete model specifications

These are the published (or reverse-engineered from model configs) specifications for major 2026-era models:

Llama 3 8B — 32 layers, $d_{\text{model}} = 4096$, 32 query heads, 8 KV heads (GQA), $d_{\text{head}} = 128$, SwiGLU FFN with $d_{\text{ff}} = 14336$, RoPE (theta=500,000), RMSNorm, 128,256 vocab, ~8.03B parameters. Trained on 15T tokens.

Llama 3 70B — 80 layers, $d_{\text{model}} = 8192$, 64 query heads, 8 KV heads (GQA), $d_{\text{head}} = 128$, SwiGLU FFN with $d_{\text{ff}} = 28672$, RoPE (theta=500,000), RMSNorm, 128,256 vocab, ~70.6B parameters. Trained on 15T tokens.

Mistral 7B — 32 layers, $d_{\text{model}} = 4096$, 32 query heads, 8 KV heads (GQA), $d_{\text{head}} = 128$, SwiGLU FFN with $d_{\text{ff}} = 14336$, sliding window attention W=4096, RoPE, RMSNorm, 32,000 vocab, ~7.3B parameters.

Mixtral 8x7B — 32 layers, $d_{\text{model}} = 4096$, 32 query heads, 8 KV heads (GQA), 8 experts per layer (top-2 routing), SwiGLU FFN with $d_{\text{ff}} = 14336$ per expert, RoPE, RMSNorm, 32,000 vocab, ~46.7B total parameters, ~12.9B active per token.

Qwen2-72B — 80 layers, $d_{\text{model}} = 8192$, 64 query heads, 8 KV heads (GQA), $d_{\text{head}} = 128$, SwiGLU FFN with $d_{\text{ff}} = 29568$, RoPE, RMSNorm, 152,064 vocab, ~72.7B parameters.

DeepSeek-V3 — 61 layers, $d_{\text{model}} = 7168$, multi-head latent attention (MLA), 256 routed experts + 1 shared expert (top-8 routing), 128,000 vocab, ~671B total, ~37B active per token.

Inspecting architectures programmatically

HuggingFace's transformers library exposes every model's configuration as a Python object. You can compare architectures without downloading the full weights:

from transformers import AutoConfig

configs = {
    "Llama-3-8B": "meta-llama/Meta-Llama-3-8B",
    "Mistral-7B": "mistralai/Mistral-7B-v0.1",
    "Qwen2-72B": "Qwen/Qwen2-72B",
}

for name, model_id in configs.items():
    cfg = AutoConfig.from_pretrained(model_id)
    print(f"\n{'='*60}")
    print(f"{name} ({model_id})")
    print(f"{'='*60}")
    print(f"  Layers:            {cfg.num_hidden_layers}")
    print(f"  d_model:           {cfg.hidden_size}")
    print(f"  Query heads:       {cfg.num_attention_heads}")
    print(f"  KV heads:          {cfg.num_key_value_heads}")
    print(f"  d_head:            {cfg.hidden_size // cfg.num_attention_heads}")
    print(f"  FFN intermediate:  {cfg.intermediate_size}")
    print(f"  Vocab size:        {cfg.vocab_size}")
    print(f"  Max position:      {cfg.max_position_embeddings}")
    print(f"  RoPE theta:        {getattr(cfg, 'rope_theta', 'N/A')}")
    print(f"  Norm type:         {getattr(cfg, 'rms_norm_eps', 'N/A')}")
    print(f"  Hidden act:        {getattr(cfg, 'hidden_act', 'N/A')}")
    print(f"  Tie embeddings:    {getattr(cfg, 'tie_word_embeddings', 'N/A')}")

Running this reveals how similar these models are at the component level — the same activation function, the same normalization, the same attention variant, the same positional encoding. The variation is in scale (how many layers, how wide) and in specific numerical choices (RoPE theta, FFN ratio, vocabulary size).

The convergence hypothesis

The uniformity of 2026 architectures suggests that the transformer design space has been thoroughly explored at current scales. The "Llama recipe" — GQA + RoPE + SwiGLU + RMSNorm + pre-norm — is a local optimum that no published ablation has meaningfully improved upon. Innovation has shifted from architecture to other axes: training data quality and composition, post-training alignment techniques (RLHF, DPO, constitutional AI), inference optimization (quantization, speculative decoding, KV cache compression), and scaling strategy (MoE, longer training, distillation).

The remaining open architecture questions are at the extremes: whether MoE or dense is better at 1T+ parameter scales, whether alternatives to softmax attention (linear attention, state-space models like Mamba) can compete at frontier quality, and whether hybrid architectures (combining attention and SSM layers) will prove optimal. None of these have produced a clear winner yet, but the base recipe — the "what every model agrees on" list — is unlikely to change without a fundamental advance.