Temperature & Sampling: Determinism vs Creativity

"Set the temperature lower for code, higher for creative writing" is good advice — but why? This article opens the box: how a model actually picks the next token, what temperature does to that choice, and how the surrounding knobs (top_p, top_k, min_p, seed) shape the trade-off between a model that's reproducible and one that's surprising.

#How sampling works

At each step a language model doesn't "decide" a word — it produces a logit (a raw score) for every token in its vocabulary. Those scores become probabilities through softmax, and then a sampler draws one token from that distribution. Everything you tune is one of two operations on that distribution:

• Reshape it — temperature makes the distribution sharper or flatter.
• Truncate it — top_k / top_p / min_p throw away the unlikely tail before sampling.

logits  ──softmax──▶  probabilities  ──temperature──▶  reshaped
        ──top_k / top_p / min_p──▶  truncated set  ──sample──▶  next token

#What temperature actually does

Temperature T divides the logits before the softmax: p_i = softmax(z_i / T). That one division is the whole story:

• T < 1 — sharpens the distribution. The top token gets even more probability; the model becomes more focused and repeatable.
• T = 1 — the model's "native" distribution, untouched.
• T > 1 — flattens the distribution. Long-shot tokens get a real chance; output gets more diverse and surprising.
• T → 0 — collapses to greedy decoding: always pick the single highest-probability token. Fully deterministic.

A worked example. Say three candidate tokens have logits 2.0, 1.0, 0.5:

Same model, same logits — only the temperature changed. Low temperature didn't make the model "smarter," it just made it commit to its top guess.

#The truncation knobs

Temperature reshapes; these filter the candidate set so the long, weird tail can't be sampled at all:

• top_k — keep only the k highest-probability tokens (e.g. 40). Hard cap on choices.
• top_p (nucleus) — keep the smallest set of tokens whose probabilities sum to p (e.g. 0.9). Adaptive: a confident step keeps few tokens, an uncertain step keeps more.
• min_p — keep tokens whose probability is at least min_p × (top token's probability). A cleaner modern alternative that scales with confidence.
• repeat_penalty — down-weights tokens already seen, to curb loops. Separate axis from temperature; don't crank both.

#Determinism and reproducibility

For anything you need to reproduce — code generation, data extraction, tool calls, evals — you want the same input to give the same output:

• Temperature 0 → greedy decoding, no randomness. Same prompt, same answer.
• Temperature > 0 with a fixed seed → randomness, but repeatable randomness — same seed reproduces the same roll.

#Why coders run cold and creatives run hot

Code has (mostly) one right answer and a syntax that punishes a single wrong token — a stray temperature spike that swaps == for = breaks the program. Creative writing is the opposite: the "wrong," low-probability token is often the interesting one. So the dial maps cleanly to the task:

#How the ADI models ship

This is exactly why theLAB's two distilled models carry different defaults in their Modelfiles. The coder ships cold; the generalist ships warm:

# adi-qwen2.5-coder-7b — coding wants determinism
PARAMETER temperature 0.3
PARAMETER top_p 0.9

# adi-qwen3.5-4b-glm5.2-general — conversation wants a little warmth
PARAMETER temperature 0.7
PARAMETER top_p 0.9

The same logic applied during training data generation: when we distilled the coder, the teacher (kimi-k2.7-code) was queried at temperature ~0.3 so its solutions stayed tight and reproducible — you want the student imitating the teacher's best answer, not a noisy one. See Picking a Teacher Model.

#Setting it in Ollama

Three places, depending on whether you want a default baked into the model or a per-request override:

# 1) baked into the model (Modelfile)
PARAMETER temperature 0.3
PARAMETER seed 42

# 2) per request via the API
curl http://localhost:11434/api/chat -d '{
  "model": "adi-qwen2.5-coder-7b-kimi2.7code-coder",
  "messages": [{"role":"user","content":"Write a binary search in Python"}],
  "options": {"temperature": 0.2, "top_p": 0.9, "seed": 42}
}'

# 3) interactively in a session
>>> /set parameter temperature 0.2