From Text to Tokens to a Forward Pass

A request to the running service carries 8,192 tokens in and brings 256 back. Before reading on, guess which side costs more. The prompt is thirty-two times longer. It is also, as this lesson will show, the cheap part.

Text becomes tokens

A model never sees text. A tokenizer splits the text into pieces drawn from a fixed vocabulary and replaces each piece with its integer id. The pieces are usually smaller than words and larger than letters: a common word is one token, a rare word is several, and a leading space normally travels with the word that follows it. Llama-3.1-8B's vocabulary has 128,256 entries.

For an inference engineer the token is the unit of everything. Context limits are counted in tokens. So is memory, as the next lesson shows, and so is the bill. The size of a request is not its character count but its token count, and the same paragraph can be a different number of tokens to two different models. Tokenization, Inside and Out covers how vocabularies are built. Here it is enough that text goes in and a list of integers comes out.

One forward pass

Hand the model a list of token ids and it does three things.

• Look up. Each id selects one row of an embedding table, turning the list of integers into a list of vectors. For Llama-3.1-8B each vector has 4,096 numbers. This step is a lookup, not arithmetic.
• Transform. The vectors pass through a stack of identical blocks, 32 of them here. Each block lets every position gather information from the positions before it, then reworks each vector on its own. Lessons 6 and 7 open a block up. For now it is the place where nearly all of the model's 8.03 billion weights are used.
• Score. A final layer turns each position's vector into one score for every entry in the vocabulary: 128,256 scores per position. The scores at the last position say which token is likely to come next.

That is a forward pass: ids in, scores out. Its defining property is that it handles every position at once. Give it 8,192 tokens and all 8,192 flow through the 32 blocks together, as one enormous batch of matrix arithmetic. This is exactly the kind of work a GPU is built for.

Generation is a loop

One forward pass yields one new token. To write a reply the model has to pick a token from those final scores, append it to the input, and run again. And again, once per token, because each token depends on the one before it. This is autoregressive generation, and it is the reason output is expensive.

On an idle server the 8,192-token prompt takes about 0.24 seconds: one pass, everything in parallel. The 256-token answer takes about 1.73 seconds: 256 passes, one after another. The prompt is 97% of the tokens and 12% of the time.

Per token the gap is starker. An input token costs about 29 microseconds. An output token costs about 6.8 milliseconds, some 235 times more. Both are computed for this model and GPU: the decode figure rests on a published measurement, the prefill figure on a model that assumes the chip sustains 60% of its peak arithmetic rate.

Choosing the token is not the model

The model's output is a list of scores, not a token. Turning scores into a choice is a separate step called sampling, and it is where temperature, top-p and top-k live. Set temperature to zero and the highest score always wins. Raise it and lower-scoring tokens get a chance. These are settings of the decoding loop, applied after the forward pass, and you can change them without touching a weight.

Two consequences come back later. Because sampling sits outside the model, an engine can intervene there, forcing valid JSON or watermarking the output, which is lesson 22. And temperature zero does not make a server's output repeatable, for a reason that has nothing to do with sampling, which is lesson 18.

Why this shapes everything after it

Hosted APIs charge more for output tokens than for input tokens, typically several times more, and now you can see why: an output token takes a whole pass to itself. What may be less obvious is why the price gap is so much smaller than 235 to 1. The answer is that a server does not run one request at a time. Many requests share each pass, and the cost of that pass is split between them. That single fact, that decode passes can be shared, is what makes serving language models affordable, and lesson 9 shows exactly where the saving comes from.

Before that, there is a problem with the loop as described. Run naively, each new pass would reprocess the entire prompt plus everything written so far. No engine does that, and the next lesson is about the cache that prevents it.