Temperature, top-p and max_tokens: the settings that shape an answer

How a model picks each word

A model writes one token at a time (a token is a word or part of a word). For each one, it scores every token it knows and turns the scores into chances. Then one token is picked, added to the text, and the process repeats.

Most of the settings below change that one step: how the next token is picked from the chances. That is why they are called sampling settings. They do not change what the model knows, only how adventurous it is when it chooses.

Temperature

Temperature reshapes the chances before a token is picked. A low temperature makes the likely tokens even more likely. A high one evens the chances out, so less likely tokens get picked more often. At 0, the model always takes its top choice.

Say the model is completing "The sky today is" and, at the default temperature of 1, gives these chances:

Low temperatures give steady, predictable answers. High ones give varied answers, and past about 1 the model starts to make choices that read as mistakes.

top_p and top_k

These cut off the unlikely tokens before one is picked. top_p keeps the smallest set of top tokens whose chances add up to p: at 0.9, it keeps tokens until they cover 90% and drops the long tail. top_k keeps a fixed number of top tokens, such as 40.

Both stop a model from picking a strange token by bad luck, while still letting it vary among good ones. top_p (also called nucleus sampling) is part of the standard OpenAI request; top_k is offered by many open-model servers but not by every API.

max_tokens: the length limit

max_tokens is the most output a model may write for one request. When it reaches the limit, it simply stops, mid-sentence if need be. It is the setting that matters most for cost, because output tokens are usually the expensive ones and a model that loops can write a lot of them.

• Set it on every request, a little above the longest answer you expect. It caps your worst case.
• Check why each answer ended. The OpenAI format reports finish_reason: "stop" for a finished answer, "length" for one cut off at the limit. A cut-off answer is incomplete, and cut-off JSON is not valid JSON.
• Models that reason before they answer count their reasoning as output. Give them room, or the limit is used up before the answer starts.

Stop sequences

A stop sequence is text that ends the answer as soon as the model writes it, and is left out of the result. It is useful when the format gives you a natural end marker: a blank line after a one-line answer, or a closing tag. It saves the tokens the model would have written past the point you care about.

Repetition penalties

Penalties make a model less likely to repeat itself. In the OpenAI format, frequency_penalty lowers a token’s chances more each time it has already appeared, and presence_penalty lowers them once it has appeared at all. Many open-model servers also offer repetition_penalty, where 1 means off.

They help with long, free-flowing text that drifts into loops. They hurt structured output: JSON repeats quotes, braces and field names by design, and a penalty pushes the model away from exactly those. For extraction and classification, leave them off.

Seed

A seed fixes the random numbers used to pick tokens, so the same request is more likely to give the same answer. More likely, not certain: the chances themselves can shift slightly between runs, because servers group requests differently and different GPUs do the arithmetic in a different order. Our guide on benchmarking explains why, and why temperature 0 does not fix it either.

Logprobs: how sure was the model?

Many APIs can return the chance the model gave each token it wrote (logprobs, short for log probabilities). For a classification, that is a useful signal: if the model put 98% on "billing", it was sure; at 40% it was guessing. In batch work you can use that to send the unsure items to a person or a bigger model, instead of checking everything.

Treat it as a hint, not a measure of accuracy. A model can be confidently wrong. Check it against a sample you have graded by hand before you rely on it.

Settings for batch work

In a chat, a person reads every answer and asks again if it is off. In a batch of 10,000 items, nobody does. That changes what good settings look like:

Prefer steady to clever

For anything code will read, a low temperature gives answers that look alike, which is what a check expects.

Always set max_tokens

One runaway answer in a chat is an annoyance. Repeated across a batch, it is a bill.

Keep settings with the prompt

Store them in the same file as the prompt and change them in the same review. An answer can only be explained if you know the settings that made it.

Change one thing at a time

Test a change on a sample you have graded, compare, and only then run it on the whole batch.

Vary settings on a retry

If an answer fails a check, asking again with a lower temperature often fixes it, and costs less than moving to a bigger model.

Settings on Otium

Otium passes your request body to the model server as you wrote it. The models run on vLLM, so the standard OpenAI settings work, and so do vLLM’s own extras, such as top_k, min_p and repetition_penalty. A request that sets no max_tokens gets a generous default limit, so a runaway answer cannot run forever; set your own to keep the cost of each item close to what you expect.

In a workflow, each step’s settings live in its body, next to its prompt, and a fallback entry can change them for a retry: the same model at a lower temperature, say, before a bigger one.

Sources and further reading