What is a token?
A large language model does not usually process text as a sequence of whole words. It works with tokens: units that may be words, pieces of words, punctuation marks or spaces attached to other text. A familiar word might fit in one token, while an unusual name might be split into several. The exact splits depend on the model’s tokenizer.
When you submit a prompt, the system converts its text into token IDs. The model uses those IDs to calculate what might come next. Its response is then converted from generated token IDs back into readable text. This is why “one token at a time” does not necessarily mean “one word at a time.”
The next-token cycle
At each step, the model considers the tokens available in its context and assigns scores to possible next tokens. Those scores are turned into probabilities. A selection process chooses a token, adds it to the response, and repeats the calculation using the expanded context. The cycle stops when the model reaches a stopping condition, such as an end-of-response token or an output limit.
Consider the prompt “Please finish this sentence: The gardener watered the”. The model might give relatively high probability to tokens that begin words such as “plants” or “flowers.” It does not need to choose the most probable option every time. Depending on the generation settings, it may select among plausible alternatives, which is one reason the same prompt can produce different wording on separate runs.
This process is sequential, but the model is not limited to the last few words. It can use earlier parts of the prompt and its own generated response, provided they remain within the context it can access. That lets it follow a requested format, continue an explanation or refer back to a detail you supplied.
Where do its predictions come from?
Before a model can generate useful text, it is trained to make predictions across many examples. Training adjusts its internal parameters so that it becomes better at predicting tokens in context. Additional training and system design may shape how an assistant responds to requests, follows instructions and handles certain kinds of content. Generation is different from training: an ordinary chat response uses the model’s existing parameters rather than updating them with each word it writes.
Next-token prediction sounds narrow, yet doing it well can require the model to represent relationships among concepts, writing styles, code patterns and parts of an argument. For example, continuing a recipe requires more than choosing words that commonly appear together: the quantities, steps and ingredients should also fit. The model can often maintain that kind of coherence, though it does not always succeed.
The context also matters. A request for a beginner-friendly definition can steer the response toward simple language; a request for a checklist can steer it toward short, actionable items. Clear instructions help because they give the model more relevant information to condition its next-token choices on.
Why fluent answers can still be wrong
The model’s immediate task is to produce a plausible continuation, not to independently verify every claim. It may generate a confident-looking explanation that contains an incorrect detail, an invented reference or an unsupported assumption. A grammatical answer is evidence that the model can produce coherent text; it is not, by itself, evidence that the answer is true.
Limited or missing context creates another source of errors. If a prompt leaves out a key requirement, the model may make an assumption instead of asking. If important information falls outside the context it can use, it may not account for it. Some systems connect models to tools or documents, but access to those resources depends on the particular setup; next-token generation alone does not provide a live view of the world.
For practical use, give the model the relevant facts and constraints, and ask it to identify uncertainties when they matter. Check consequential details against appropriate sources or your own records. Thinking of an answer as a generated draft—rather than a verified finding—makes it easier to use the model’s speed without mistaking fluency for accuracy.
