How LLMs process prompts

A prompt enters the model as a string of characters and exits as a sequence of integer token IDs. The tokenizer splits raw text into subword units using a learned vocabulary, maps each unit to an ID, and produces the actual input to the model. Every subsequent operation — attention, position encoding, generation — operates on these tokens, not on characters or words. Understanding the tokenizer is prerequisite to understanding prompt behavior.

Tokenization: BPE and SentencePiece

Byte Pair Encoding (Sennrich et al. 2016) builds a vocabulary by iteratively merging the most frequent pair of adjacent symbols in a training corpus. Starting from individual bytes (or characters), the algorithm identifies the most common bigram, adds it as a new vocabulary entry, replaces all occurrences in the corpus, and repeats for a fixed number of merges. GPT-4o uses a BPE vocabulary of ~200,000 tokens (cl200k_base). Claude uses a similar-sized BPE vocabulary trained on a different corpus.

SentencePiece (Kudo & Richardson 2018) operates directly on raw text without pre-tokenization — no language-specific word splitting. It treats the input as a sequence of Unicode characters (or bytes) and learns a subword vocabulary using either BPE or a unigram language model. Llama models use SentencePiece with byte-fallback, meaning any byte sequence can be represented even if it is not in the vocabulary.

The practical consequence: tokenization is not deterministic across models. The same English sentence tokenizes differently under GPT-4o (cl200k_base) versus Claude versus Llama 3. Code tokenizes very differently from prose — whitespace-heavy Python may use 2-3x more tokens than the equivalent natural language description.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

text = "Prompt engineering requires understanding tokenization."
tokens = enc.encode(text)
print(f"Text: {text}")
print(f"Token count: {len(tokens)}")
print(f"Token IDs: {tokens}")
print(f"Decoded tokens: {[enc.decode([t]) for t in tokens]}")

code = """def calculate_cost(tokens, model="gpt-4o"):
    price_per_million = 2.50
    return (tokens / 1_000_000) * price_per_million"""

code_tokens = enc.encode(code)
print(f"\nCode token count: {len(code_tokens)}")
print(f"Characters: {len(code)}")
print(f"Chars per token: {len(code) / len(code_tokens):.1f}")

English prose averages ~4 characters per token on GPT-4o's tokenizer. Code averages ~3 characters per token due to formatting characters, indentation, and operator sequences that do not merge well. JSON is worse: ~2.5 characters per token because of repeated structural characters ({, }, ", :) that remain as individual tokens.

Context windows and positional encoding

The context window is the maximum number of tokens the model can process in a single forward pass. This is a hard architectural limit, not a soft degradation:

GPT-4o: 128K tokens (~96K words)
Claude 3.5 Sonnet: 200K tokens (~150K words)
Gemini 1.5 Pro: 2M tokens (~1.5M words)
Llama 3.1 405B: 128K tokens

Positional information enters through positional encodings. The original Transformer (Vaswani et al. 2017) used fixed sinusoidal encodings. Modern models use Rotary Position Embeddings (RoPE, Su et al. 2021), which encode relative position directly into the attention score computation by rotating query and key vectors. RoPE allows extrapolation beyond training-time context lengths — with techniques like YaRN (Peng et al. 2023), models trained at 4K context can be extended to 128K+ with fine-tuning on a small amount of long-context data.

The attention mechanism computes all-pairs interactions between tokens: token at position i attends to every other token with a weight determined by the query-key dot product. Positional encoding modulates these scores so the model can distinguish "token A at position 5" from "token A at position 500." Without positional encoding, attention is permutation-invariant — the model cannot tell word order.

The "lost in the middle" phenomenon

Liu et al. (2023) demonstrated that language models exhibit a U-shaped attention pattern: information placed at the beginning or end of the context window is retrieved more reliably than information placed in the middle. In a multi-document QA task with 20 documents, accuracy was 56% when the relevant document was first, 42% when it was in position 10 (middle), and 52% when it was last.

This effect is robust across model sizes (4B to 70B parameters) and persists even in models trained with long-context objectives. The mechanism is partially explained by the attention sink phenomenon (Xiao et al. 2023): the first few tokens receive disproportionate attention regardless of content, acting as a "sink" for attention mass that the model has learned to use as a no-op.

Practical implications for prompt design:

Place critical instructions at the beginning of the prompt (system prompt position)
Place the query or question at the end of the prompt
If including multiple documents or examples, put the most important ones first or last
For retrieval-augmented generation, re-rank retrieved documents so the most relevant is not buried in position 5 of 10

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

def measure_prompt_positions(system: str, documents: list[str], query: str) -> dict:
    """Measure token positions of each component in a prompt."""
    full_prompt = system + "\n\n" + "\n\n".join(documents) + "\n\n" + query
    total_tokens = len(enc.encode(full_prompt))

    positions = {}
    running = 0
    for i, doc in enumerate(documents):
        doc_tokens = len(enc.encode(doc))
        positions[f"doc_{i}"] = {
            "start_token": running,
            "end_token": running + doc_tokens,
            "relative_position": running / total_tokens
        }
        running += doc_tokens

    positions["total_tokens"] = total_tokens
    positions["query_position"] = running / total_tokens
    return positions

system = "You are a research assistant. Answer questions based on the provided documents."
docs = [f"Document {i}: " + "x " * 200 for i in range(10)]
query = "Based on the documents above, what is the main finding?"

positions = measure_prompt_positions(system, docs, query)
for key, val in positions.items():
    if key.startswith("doc_"):
        print(f"{key}: position {val['relative_position']:.1%} "
              f"(tokens {val['start_token']}-{val['end_token']})")
print(f"Total tokens: {positions['total_tokens']}")
print(f"Query at: {positions['query_position']:.1%}")

Token counting and cost calculation

API pricing is per-token, split between input (prompt) and output (completion). As of mid-2025:

GPT-4o: $2.50/M input, $10.00/M output
Claude 3.5 Sonnet: $3.00/M input, $15.00/M output
GPT-4o-mini: $0.15/M input, $0.60/M output
Claude 3.5 Haiku: $0.80/M input, $4.00/M output

Output tokens are 3-4x more expensive than input tokens because generation is autoregressive — each output token requires a full forward pass through the model (with KV cache reuse), while input tokens are processed in a single parallel pass.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

def estimate_cost(
    prompt: str,
    estimated_output_tokens: int,
    model: str = "gpt-4o"
) -> dict:
    """Estimate API cost for a single request."""
    pricing = {
        "gpt-4o": {"input": 2.50, "output": 10.00},
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
        "claude-3.5-sonnet": {"input": 3.00, "output": 15.00},
        "claude-3.5-haiku": {"input": 0.80, "output": 4.00},
    }

    input_tokens = len(enc.encode(prompt))
    prices = pricing[model]

    input_cost = (input_tokens / 1_000_000) * prices["input"]
    output_cost = (estimated_output_tokens / 1_000_000) * prices["output"]

    return {
        "input_tokens": input_tokens,
        "output_tokens": estimated_output_tokens,
        "input_cost_usd": input_cost,
        "output_cost_usd": output_cost,
        "total_cost_usd": input_cost + output_cost,
        "cost_per_1k_requests": (input_cost + output_cost) * 1000
    }

system_prompt = "You are a classification assistant." * 50  # ~250 tokens
user_message = "Classify the following customer email: " + "word " * 500  # ~500 tokens
full_prompt = system_prompt + "\n" + user_message

result = estimate_cost(full_prompt, estimated_output_tokens=50, model="gpt-4o")
print(f"Input tokens:  {result['input_tokens']:,}")
print(f"Output tokens: {result['output_tokens']:,}")
print(f"Input cost:    ${result['input_cost_usd']:.6f}")
print(f"Output cost:   ${result['output_cost_usd']:.6f}")
print(f"Total cost:    ${result['total_cost_usd']:.6f}")
print(f"Cost per 1K requests: ${result['cost_per_1k_requests']:.4f}")

The prompt as a probabilistic program

A prompt is a program that executes on a probabilistic computer. The model's forward pass is deterministic given a fixed random seed (via the temperature parameter), but the mapping from prompt to output is sensitive to small perturbations in a way that deterministic programs are not. Changing a single word in a prompt can shift output probabilities by 20-30% on classification tasks (Lu et al. 2022).

The prompt specifies:

State — the context window contents at inference time (system prompt + user input + any retrieved context)
Constraints — what the model should and should not do (instruction following)
Format — the expected structure of the output (JSON, markdown, free text)
Priors — few-shot examples that shift the model's output distribution toward desired behavior

Each component competes for a finite budget (the context window). A 128K context window is large but not unlimited — a system prompt of 2K tokens, 10 retrieved documents of 1K tokens each, conversation history of 5K tokens, and a user message of 500 tokens leaves 110.5K tokens for output and headroom. In practice, you want to keep the context well under the maximum to avoid degradation near the boundary and to control costs.

The key mental model: the prompt is not a request to an assistant. It is a configuration of a text-completion machine. The system prompt sets the initial state. The user message sets the continuation target. The model's job is to produce the most probable continuation given the full context — and your job as a prompt engineer is to construct a context that makes the desired output the most probable continuation.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

def context_budget(
    context_limit: int,
    system_tokens: int,
    history_tokens: int,
    retrieval_tokens: int,
    user_tokens: int,
    reserved_output: int = 4096
) -> dict:
    """Calculate remaining context budget after allocating components."""
    used = system_tokens + history_tokens + retrieval_tokens + user_tokens
    available = context_limit - used - reserved_output
    utilization = used / context_limit

    return {
        "context_limit": context_limit,
        "used_tokens": used,
        "reserved_output": reserved_output,
        "available_tokens": available,
        "utilization": utilization,
        "breakdown": {
            "system": f"{system_tokens:,} ({system_tokens/context_limit:.1%})",
            "history": f"{history_tokens:,} ({history_tokens/context_limit:.1%})",
            "retrieval": f"{retrieval_tokens:,} ({retrieval_tokens/context_limit:.1%})",
            "user": f"{user_tokens:,} ({user_tokens/context_limit:.1%})",
            "output_reserve": f"{reserved_output:,} ({reserved_output/context_limit:.1%})",
        }
    }

budget = context_budget(
    context_limit=128_000,
    system_tokens=1_500,
    history_tokens=8_000,
    retrieval_tokens=12_000,
    user_tokens=500,
    reserved_output=4_096
)

print("Context budget breakdown:")
for component, value in budget["breakdown"].items():
    print(f"  {component:20s} {value}")
print(f"\n  Available headroom:  {budget['available_tokens']:,} tokens")
print(f"  Utilization:         {budget['utilization']:.1%}")

The token budget is the fundamental constraint of prompt engineering. Every design decision — how verbose the system prompt is, how many examples to include, how much history to keep, how many documents to retrieve — is a tradeoff within this fixed budget. Treating the context window as an information architecture problem, not a text composition problem, is the shift from writing prompts to engineering them.