Understand the algorithm that turns a trained language model into text, well enough to say what every sampling setting does to the distribution and to tell a model failure apart from a decoder failure

How a Model Picks Its Next Word

The model produces a distribution and something else has to choose a word. This course covers that choice, from greedy search and beam width through temperature, truncation, repetition, grammars and speculative decoding.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

This is course 4 of 8 in a reading track: explain the model everybody is actually using. The track is an order, not a gate, and the rest of it is on one page.

01The Model Does Not Write AnythingA language model produces one number per token in its vocabulary and stops. Everything about which word appears is decided afterwards, by a separate algorithm with its own settings.
02Two Different Questionsopening onlyTaking the best token at every step answers one question, and finding the best sequence answers another. They have different answers, and the gap between them is where search algorithms live.
03What a Beam Keeps, and What It Costsopening onlyBeam search is greedy selection applied to sets instead of tokens. Writing its score out reveals a bias towards short sequences, and the standard repair introduces a bias of its own.
04What Temperature Actually Doesopening onlyDividing the scores by a constant before the softmax rescales every log-ratio between tokens by the same factor. Both limits are worth knowing, and neither end is creative.
05Throwing Away the Worst Optionsopening onlyThe tail of the distribution holds almost no probability and almost every token. Sampling without removing it is the main reason generated text goes wrong, and there are two standard ways to cut.
06The Loop Is the Most Probable Continuationopening onlyRepetition is not a bug in the decoder. Under the model, a phrase already written is more likely to be written again, so the most probable continuation of a repeated phrase is another copy of it.
07Making the Output Fit a Formatopening onlyIf the answer has to be valid JSON, or one of four labels, the reliable way is not to ask nicely. It is to set the probability of every token that would break the format to zero before sampling.
08Guessing Ahead Without Changing the Answeropening onlyA small model can guess several tokens ahead and the large one checks them all in a single pass. The accept rule is built so that the text that comes out is distributed exactly as before.