Generation Settings and System Prompts

Build the mental model

Every request you send to a local model is really two things bundled together: the conversation itself, structured as a list of messages, and a set of generation settings that control how the model picks its next word. The message list conventionally has three roles.

• System message — sets the model's persona and rules before the conversation starts, for example “You are a patient programming tutor who explains ideas in Burmese and English.”
• User message — what a person actually typed.
• Assistant message — the model's own previous replies, kept in the list so it has memory of the conversation so far.

Layered on top of this structure are the generation settings, sometimes called sampling parameters, which shape how the model turns its internal probabilities into an actual word.

• Temperature — scales how much the model favors its single most likely next word versus spreading probability across less likely ones: near zero it behaves almost greedily, picking the safest word; higher values let rarer, more surprising words compete, producing more varied output.
• Top-p (nucleus sampling) — restricts the model to the smallest set of candidate words whose combined probability crosses a threshold like 0.9, cutting off the long tail of very unlikely words.
• Top-k — a simpler variant that only considers the k highest-probability words at each step.
• Max tokens — simply caps how long a reply can run before it is cut off.

Temperature

A generation setting that scales how strongly the model favors its single most likely next word versus spreading probability across less likely ones — low values produce focused, predictable output; high values produce more varied output.

Top-p

Nucleus sampling: restricts the model to the smallest set of candidate words whose combined probability crosses a threshold (like 0.9), cutting off the long tail of very unlikely words.

System Prompt

A message set before the conversation starts that defines the model's persona, rules, or behavior for the rest of the exchange, without the end user typing or seeing it.

PROMPT STRUCTURE AND GENERATION DIALS
-------------------------------------
------------------------------------------------------------
  SYSTEM PROMPT  ---+
  "You are a         |
   tutor..."         |
                      v
  USER PROMPT  --> [ MODEL ] --> ASSISTANT RESPONSE
  "Explain              ^
   for loops"           |
                 dials turned before generating:
                 -----------------------------
                 temperature  (focused <-> varied)
                 top-p        (nucleus cutoff)
                 top-k        (candidate pool size)
                 max tokens   (reply length cap)

Connect it to a real scenario

Suppose you're building the Burmese programming tutor mentioned earlier in this lesson. You'd start with a system message like “You are a patient programming tutor who explains ideas in simple Burmese and English, and always includes a short code example.” That single sentence shapes every reply for the rest of the conversation, without the user ever seeing it.

• Everyday Q&A — you'd set temperature low, maybe 0.2, and a modest top-p like 0.9, so the model consistently reaches for its most confident, textbook-style phrasing instead of wandering into creative but possibly misleading explanations.
• Practice-exercise generation — so learners don't see the same example twice, you'd raise temperature to something like 0.9 for that specific request only, while keeping the tutoring conversation itself on low temperature.

This is the practical lesson: temperature isn't one global setting for your whole app, it's a per-request dial you choose based on whether you want reliability or variety for that particular call, and the code exercise below has you build exactly this kind of request structure yourself.

• A chat request bundles a message list (system/user/assistant roles) with generation settings that shape how the model samples the next word.
• Temperature, top-p, and top-k are different knobs for the same tradeoff: focused/predictable versus varied/creative output.
• Generation settings are a per-request choice, not one global setting for your whole app.

Try the working example

import json


def build_chat_messages(system_prompt, conversation_history, user_message):
    messages = [{"role": "system", "content": system_prompt}]
    for turn in conversation_history:
        messages.append({"role": "user", "content": turn["user"]})
        messages.append({"role": "assistant", "content": turn["assistant"]})
    messages.append({"role": "user", "content": user_message})
    return messages


if __name__ == "__main__":
    system_prompt = (
        "You are a patient programming tutor who explains ideas in "
        "simple Burmese and English, with short code examples."
    )
    history = [
        {
            "user": "Python list ဆိုတာ ဘာလဲ?",
            "assistant": "List ဆိုတာ item အများကြီးကို စဉ်လိုက် သိမ်းထားနိုင်တဲ့ container တစ်ခုပါ.",
        }
    ]
    new_message = "for loop နဲ့ list ထဲက item တွေကို ဘယ်လို ထုတ်ကြည့်လဲ?"

    messages = build_chat_messages(system_prompt, history, new_message)
    print(json.dumps(messages, ensure_ascii=False, indent=2))

Running the script prints a JSON array of four message objects: the system message defining the tutor persona, the first user message asking what a Python list is, the assistant's stored reply about lists, and the new user message asking how to loop over a list with a for loop — each with 'role' and 'content' keys, formatted with 2-space indentation.

Ollama API documentation (chat request options) — Local AI / Local LLM