Encoder-decoder vs. decoder-only

The original transformer (Vaswani et al. 2017) has two separate stacks of transformer blocks connected by cross-attention. The encoder processes the full input sequence with bidirectional attention — no causal mask, every token attends to every other token in the input. The decoder generates output tokens autoregressively with causal self-attention, and at every layer cross-attends to the encoder's final output. This design was built for machine translation, where the input (source sentence) and output (target sentence) have fundamentally different roles.

The encoder-decoder architecture in detail

The encoder stack is a series of blocks, each containing bidirectional self-attention and a feed-forward network. "Bidirectional" means the attention mask is all-ones: token 3 in the input can attend to tokens 1, 2, 4, 5, and every other input token. This allows each token's representation to be informed by the full surrounding context — both left and right.

The decoder stack has three sublayers per block instead of two: (1) causal self-attention over previously generated tokens, (2) cross-attention where the decoder queries attend to encoder keys and values, (3) a feed-forward network. The cross-attention is the architectural bridge — it's how the decoder "reads" the encoded input.

import torch
import torch.nn as nn
import torch.nn.functional as F


class EncoderBlock(nn.Module):
    def __init__(self, d_model, n_heads, d_ff):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model),
        )
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)

    def forward(self, x):
        # Bidirectional: no attention mask
        attn_out, _ = self.self_attn(x, x, x)
        x = self.norm1(x + attn_out)
        x = self.norm2(x + self.ffn(x))
        return x

class DecoderBlock(nn.Module):
    def __init__(self, d_model, n_heads, d_ff):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
        self.cross_attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model),
        )
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.norm3 = nn.LayerNorm(d_model)

    def forward(self, x, encoder_output, causal_mask=None):
        # Masked self-attention (causal)
        attn_out, _ = self.self_attn(x, x, x, attn_mask=causal_mask)
        x = self.norm1(x + attn_out)
        # Cross-attention: decoder queries, encoder keys and values
        cross_out, _ = self.cross_attn(x, encoder_output, encoder_output)
        x = self.norm2(x + cross_out)
        # Feed-forward
        x = self.norm3(x + self.ffn(x))
        return x

class EncoderDecoder(nn.Module):
    def __init__(self, src_vocab, tgt_vocab, d_model, n_heads, d_ff,
                 n_enc_layers, n_dec_layers):
        super().__init__()
        self.encoder_embed = nn.Embedding(src_vocab, d_model)
        self.decoder_embed = nn.Embedding(tgt_vocab, d_model)
        self.encoder_layers = nn.ModuleList([
            EncoderBlock(d_model, n_heads, d_ff) for _ in range(n_enc_layers)
        ])
        self.decoder_layers = nn.ModuleList([
            DecoderBlock(d_model, n_heads, d_ff) for _ in range(n_dec_layers)
        ])
        self.lm_head = nn.Linear(d_model, tgt_vocab)

    def encode(self, src_ids):
        x = self.encoder_embed(src_ids)
        for layer in self.encoder_layers:
            x = layer(x)
        return x

    def decode(self, tgt_ids, encoder_output):
        T = tgt_ids.shape[1]
        causal_mask = torch.triu(
            torch.ones(T, T, device=tgt_ids.device), diagonal=1
        ).bool()

        x = self.decoder_embed(tgt_ids)
        for layer in self.decoder_layers:
            x = layer(x, encoder_output, causal_mask)
        return self.lm_head(x)

    def forward(self, src_ids, tgt_ids):
        enc_out = self.encode(src_ids)
        return self.decode(tgt_ids, enc_out)

The cross-attention layer has its own Q, K, V projections (3 × d_model² parameters per decoder layer), adding roughly 50% more parameters per decoder block compared to a decoder-only block. For the original transformer-base (d_model=512, d_ff=2048, 6+6 layers): encoder has ~44M parameters, decoder has ~66M (due to cross-attention), total ~110M. A decoder-only model with the same per-block budget would fit ~8 layers instead of 6+6.

Encoder-only: BERT

BERT (Devlin et al. 2019) removes the decoder entirely. It's a stack of encoder blocks — bidirectional self-attention with no causal mask — with task-specific heads instead of autoregressive generation.

Training objective: masked language modeling (MLM). Randomly mask 15% of input tokens (80% replaced with [MASK], 10% replaced with a random token, 10% left unchanged), then predict the original token at each masked position from the full bidirectional context.

BERT-base: 12 layers, d_model=768, 12 heads, d_ff=3072. 110M parameters.

BERT-large: 24 layers, d_model=1024, 16 heads, d_ff=4096. 340M parameters.

The bidirectional context makes BERT excellent for understanding tasks — classification, named entity recognition, semantic similarity, extractive question answering — because the representation of each token incorporates information from both its left and right context. The token "bank" in "river bank" attends to "river" regardless of word order; in a causal model, "bank" can only attend to "river" if "river" appears first.

But BERT cannot generate text autoregressively. Autoregressive generation requires predicting token t from tokens 1, ..., t-1 only. BERT's training objective and attention pattern are bidirectional — the model has never learned to predict from left-context alone, and its representations at each position depend on future tokens that wouldn't exist during generation. You cannot simply "run BERT left-to-right" — the architecture would need a causal mask (changing the attention pattern) and retraining from scratch (changing the learned representations).

Decoder-only: GPT, Llama, Claude, Mistral

The decoder-only architecture removes the encoder and the cross-attention sublayers. What remains is a single stack of blocks, each with causal self-attention and an FFN — structurally identical to the decoder half of the original transformer, but without any conditioning on a separate encoder output.

class DecoderOnly(nn.Module):
    def __init__(self, vocab_size, d_model, n_heads, n_kv_heads, d_ff, n_layers):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, d_model)
        self.layers = nn.ModuleList([
            TransformerBlock(d_model, n_heads, n_kv_heads, d_ff)
            for _ in range(n_layers)
        ])
        self.norm = RMSNorm(d_model)
        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)

    def forward(self, input_ids):
        B, T = input_ids.shape
        x = self.embed(input_ids)
        mask = torch.tril(torch.ones(T, T, device=x.device)).unsqueeze(0).unsqueeze(0)
        for layer in self.layers:
            x = layer(x, mask)
        return self.lm_head(self.norm(x))

Training objective: causal language modeling. At each position t, predict token t+1 given tokens 1, ..., t. The loss is the cross-entropy between the predicted distribution and the actual next token, averaged across all positions and all sequences in the batch. This is conceptually simpler than MLM — there's no masking strategy to design, no special tokens, and the model naturally generates text by iteratively sampling from its own predictions.

Why decoder-only dominates generative AI

Four structural advantages explain why every frontier language model since 2022 is decoder-only:

Simpler architecture, fewer hyperparameters. One stack instead of two. No cross-attention sublayers (which add 3 weight matrices per decoder layer — cross Q, K, V — plus an output projection). No decision about encoder depth vs. decoder depth. No question of whether to share embeddings between encoder and decoder. Fewer moving parts means faster iteration, easier debugging, and simpler distributed-training implementations. The entire Llama 3 block is ~60 lines of PyTorch; an encoder-decoder block with cross-attention is ~90 lines with more complex data flow.

Unified training objective on arbitrary text. Next-token prediction works on any sequence of tokens. You can train on books, code, conversations, HTML, JSON, LaTeX — all with the same loss function, no preprocessing beyond tokenization. Encoder-decoder models require either parallel data (source-target pairs, as in translation) or a constructed denoising objective (mask-then-reconstruct, as in T5/BART). Causal LM training on web-scale corpora of trillions of tokens doesn't need curated parallel pairs — just text.

Parameter efficiency during inference. In an encoder-decoder architecture, the encoder runs once on the input, then the decoder runs autoregressively for each output token. During the generation phase (which dominates inference cost for long outputs), only the decoder's parameters are actively computing — the encoder's parameters sit in memory unused. If the encoder and decoder are equal-sized (e.g., T5's 12+12 layers), half the model's parameters contribute nothing during generation. In decoder-only, every parameter is used at every step.

KV cache simplicity. During autoregressive generation, decoder-only models maintain one KV cache: the self-attention keys and values for all previously processed tokens. This cache grows linearly with sequence length. Encoder-decoder models maintain two caches: (a) the self-attention KV cache (growing with output length), and (b) the cross-attention KV cache (fixed after encoding, but occupying n_dec_layers × src_len × d_model memory). Managing two separate caches complicates memory planning, batching strategies (vLLM's PagedAttention, for instance), and serving infrastructure. For systems processing thousands of concurrent requests, this complexity translates directly to engineering cost and latency.

Where encoder-decoder persists

Encoder-decoder architectures remain the right choice when the task has a fundamental asymmetry between input and output modalities or processing patterns:

T5 (Raffel et al. 2020): 11B parameters at its largest. Google's text-to-text framework that casts every NLP task as "input text → output text." The encoder processes the full input bidirectionally (e.g., a document to summarize); the decoder generates the output (e.g., the summary). T5 is still used internally at Google for structured extraction tasks where bidirectional input understanding provides measurable benefit over causal-only prompting.

Whisper (Radford et al. 2023, OpenAI): speech-to-text. The encoder processes an 80-channel log-mel spectrogram (30 seconds of audio, represented as a 1500 × 80 matrix). Audio and text are fundamentally different modalities — the encoder uses a convolutional frontend followed by transformer blocks, which wouldn't make sense in an autoregressive decoder. The decoder generates the transcript token by token, cross-attending to the audio representation.

NLLB (No Language Left Behind, Meta, 2022): 200-language machine translation with 54B parameters. The structural separation between source language (encoder) and target language (decoder) maps naturally onto the encoder-decoder architecture and allows the model to learn language-specific representations in each stack.

The prefix LM hybrid

PaLM (Chowdhery et al. 2022) and UL2 (Tay et al. 2022) explore a middle ground: the prefix language model. The model is architecturally decoder-only (single stack), but during training, a prefix portion of the input is attended to bidirectionally — the causal mask is removed for the prefix positions, while the rest is generated causally.

def create_prefix_lm_mask(seq_len, prefix_len):
    mask = torch.zeros(seq_len, seq_len)
    # Prefix positions attend to all other prefix positions (bidirectional)
    mask[:prefix_len, :prefix_len] = 1
    # Generation positions attend causally
    for i in range(prefix_len, seq_len):
        mask[i, :i+1] = 1
    return mask

# Example: input is 10 tokens, first 6 are prefix (bidirectional), last 4 are generated
mask = create_prefix_lm_mask(seq_len=10, prefix_len=6)
# Prefix tokens (0-5) see each other bidirectionally
# Generation tokens (6-9) see all prefix + previous generation tokens causally

This gives encoder-like bidirectional understanding of the input context (the prefix) while maintaining a single architecture that can also generate autoregressively. The advantage over true encoder-decoder: simpler implementation, no cross-attention overhead, no separate caches, and the ability to vary the prefix boundary per example without architectural changes. The disadvantage: the prefix still consumes decoder capacity (its representations pass through the full stack), whereas in encoder-decoder the encoder can be smaller or differently structured than the decoder.

The numbers that settled the debate

GPT-3 (Brown et al. 2020) was the inflection point. A 175B decoder-only model trained on 300B tokens of generic internet text could few-shot virtually any NLP task — translation, summarization, QA, arithmetic — without task-specific fine-tuning. T5-11B, the largest encoder-decoder model at the time, required fine-tuning on each task to approach GPT-3's few-shot performance. The implication was clear: a large enough decoder-only model with enough training data learns to be its own encoder, its own task router, and its own decoder simultaneously.

The scaling law findings reinforced this. Kaplan et al. 2020 and Hoffmann et al. 2022 ("Chinchilla") demonstrated that decoder-only models follow clean, predictable power laws — more compute yields predictably lower loss across all benchmarks. The field invested billions of dollars of compute into validating and exploiting these scaling laws, and all of that investment assumed the decoder-only architecture. Encoder-decoder scaling received comparatively little study because by 2021, the momentum had shifted irreversibly.

The infrastructure followed the architecture. FlashAttention (Dao et al. 2022), PagedAttention/vLLM (Kwon et al. 2023), tensor parallelism libraries (Megatron-LM), and the entire serving ecosystem (TGI, TensorRT-LLM) are all optimized primarily for decoder-only causal attention patterns. Building an encoder-decoder at frontier scale (400B+ parameters) would require re-engineering these systems to handle the asymmetric compute pattern of cross-attention — a cost no lab has chosen to pay when decoder-only models demonstrably work.

The result: every frontier language model in 2024–2026 — GPT-4 (OpenAI), Claude 3/4 (Anthropic), Llama 3/3.1 (Meta), Gemini (Google), Mistral Large (Mistral AI), Qwen 2.5 (Alibaba), DeepSeek-V3 (DeepSeek) — is decoder-only. Encoder-decoder persists only where cross-modal structural separation provides genuine, measurable benefit that the decoder-only formulation cannot replicate through prompting alone.