Tokens, Sampling & Context

By the end of this session you will be able to:

• Read the probability the model assigned to every token it just produced, instead of guessing why it said what it said.
• Predict what temperature, top-p and top-k do to a given distribution, and set them on purpose rather than by superstition.
• Work out what a request actually costs in context, and explain why quality drops long before the window is full.

A model only ever produces one thing: a distribution

A causal language model does exactly one job. Given a sequence of token IDs, it emits a vector of scores, one per vocabulary entry, for the token that comes next. Everything else, sampling, streaming, chat, agents, is a loop wrapped around that single step.

Two consequences follow, and both bite people who skip this part.

First, the model does not see your text. It sees integers. Qwen/Qwen2.5-0.5B-Instruct has a vocabulary of 151,936 entries, and the tokenizer decides how your string maps onto them. Look at it:

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
text = "The strawberry costs 1500 riyals."
ids = tok(text).input_ids
print(len(ids), ids)
print(tok.convert_ids_to_tokens(ids))

You will see the leading space folded into the token after it, the number split into pieces that have nothing to do with place value, and a rare word broken into fragments. This is why models miscount letters in a word, why arithmetic on long numbers is unreliable, and why a prompt in Arabic or Hebrew can cost three times the tokens of the same sentence in English. None of that is a reasoning failure. The model never received the units you thought you sent.

Second, "the model is confident" is a measurable quantity, not a vibe. You can read it.

Reading the distribution instead of guessing at it

Set return_dict_in_generate=True and output_scores=True, then use compute_transition_scores to get the log probability of each token the model actually selected.

import numpy as np
from transformers import AutoModelForCausalLM, AutoTokenizer
 
name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, device_map="auto")
 
inputs = tok(["The capital of France is"], return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=8,
    do_sample=False,
    return_dict_in_generate=True,
    output_scores=True,
)
 
scores = model.compute_transition_scores(out.sequences, out.scores, normalize_logits=True)
generated = out.sequences[:, inputs.input_ids.shape[1]:]
 
for token_id, score in zip(generated[0], scores[0]):
    prob = np.exp(score.item())
    print(f"{tok.decode(token_id)!r:12} logprob={score.item():7.3f}  p={prob:.2%}")

Run it and you get a per-token confidence trace. Function words and the obvious continuation land above 80 percent. The token where the model committed to a claim, a name, a number, a date, is usually the one sitting at 20 or 30 percent. That token is where your hallucination lives, and you can now point at it.

Note normalize_logits=True. Without it you get raw, unnormalized scores that are not log probabilities, and np.exp on them gives you nonsense. Also note do_sample=False. Greedy decoding is the default in Transformers, and if you set temperature without do_sample=True the library warns you and ignores it.

Temperature, top-p and top-k are three different knobs

They all shape the same distribution, but at different stages, and confusing them is the most common source of "I turned temperature down and it still rambles".

Temperature divides the logits before the softmax. Below 1.0 it sharpens the distribution, pushing mass onto tokens that were already ahead. Above 1.0 it flattens it, giving the long tail a real chance. It changes the shape of the whole curve and removes nothing.

Top-k truncates to the k highest-probability tokens, then renormalizes. Fixed count, regardless of shape. The Transformers default is 50.

Top-p, or nucleus sampling, sorts by probability and keeps the smallest set whose mass sums to at least p, then renormalizes. Adaptive: where the model is confident, that might be two tokens; where it is unsure, forty.

Order matters. Transformers applies temperature first, then top-k, then top-p. So a high temperature with a tight top-p is not a contradiction. You are reweighting inside the nucleus while still refusing to leave it, and that is usually what people actually want when they say "creative but not unhinged".

prompt = tok(["Three uses for a paperclip:"], return_tensors="pt").to(model.device)
prompt_len = prompt.input_ids.shape[1]
 
for temperature, top_p in [(0.0, 1.0), (0.7, 0.95), (1.5, 1.0), (1.5, 0.9)]:
    kwargs = {"max_new_tokens": 40}
    if temperature == 0.0:
        kwargs["do_sample"] = False
    else:
        kwargs.update(do_sample=True, temperature=temperature, top_p=top_p, top_k=0)
    out = model.generate(**prompt, **kwargs)
    text = tok.batch_decode(out[:, prompt_len:], skip_special_tokens=True)[0]
    print(f"T={temperature} top_p={top_p} -> {text.strip()[:120]}")

The (1.5, 1.0) row is the one to read closely. With the nucleus open and the distribution flattened, the model will eventually pick a token from deep in the tail, and everything after that point is conditioned on a mistake it cannot take back. The (1.5, 0.9) row stays varied but coherent. That difference is the entire argument for nucleus sampling.

There is no true temperature 0. Passing temperature=0 is not valid sampling; what you want is do_sample=False. And greedy decoding is still not deterministic across batch sizes, hardware or dtype, because floating point reduction order changes. For a fixed setup, transformers.set_seed(0) before each call makes sampled runs reproducible.

The context window is a budget, not a container

Every model config states a maximum position count:

from transformers import AutoConfig
 
cfg = AutoConfig.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
print(cfg.max_position_embeddings, cfg.num_hidden_layers, cfg.num_key_value_heads)

That prints 32768 for this model. Treating it as a container, "I have 32k, I will fill it", is where cost and quality both go wrong.

Attention is quadratic in sequence length for the prefill, and the KV cache you must hold in memory grows linearly with every token:

kv_bytes = 2 * layers * kv_heads * head_dim * tokens * bytes_per_element

The 2 is keys plus values. Double your prompt and you double resident memory per request and roughly quadruple prefill compute. That is the cost side.

The quality side is worse, because it is silent. Retrieval accuracy inside a long context is not uniform across positions: material at the very start and very end is used far more reliably than material in the middle. Nothing errors. The model just quietly stops using the paragraph you buried at token 12,000. If your answers degrade as you add context, adding more context is not the fix.

Try it

Take one prompt from something you actually care about and instrument it.

1. Generate 60 tokens greedily with output_scores=True and print the per-token probability using the code above.
2. Find every generated token below 40 percent and write down what each one is.
3. Now regenerate the same prompt five times at temperature=1.0, top_p=1.0, and five times at temperature=1.0, top_p=0.9, with set_seed varied each run.

Success condition: you can name the specific low-probability token where the ten sampled runs first diverge from each other, and you can state whether the top_p=0.9 runs diverged at the same place or later. If they diverged later, you have just measured what nucleus sampling bought you on your own task.

Common mistakes

• Setting temperature without do_sample=True. Generation stays greedy, Transformers logs a warning you did not read, and you conclude temperature does nothing. Set the strategy first, the knobs second.
• Treating temperature=0 as a supported setting. It is a division by zero dressed up by API wrappers. Use do_sample=False and say greedy when you mean greedy.
• Tuning temperature and top-p together in one sweep. Fix top-p, sweep temperature, then fix temperature and sweep top-p. Two knobs moving at once on a stochastic output tells you nothing.
• Counting characters or words when you mean tokens. Budget in tokens, from the tokenizer, for the exact model. A 4-characters-per-token rule of thumb is wrong by 3x on non-English text and on code.
• Blaming the model for a tokenizer artifact. Before debugging a counting or arithmetic failure, print convert_ids_to_tokens on the input. Half the time the answer is right there.

Where this goes next

You have been feeding this instruct model raw strings, which is not the format it was trained on. Next session, Chat Templates, you apply the tokenizer's own template, then deliberately use a mismatched one and watch output quality fall apart without a single error being raised.