Skip to content
Optimal Lab

AI Engineering · interactive

System, user and assistant messages

Chat APIs take a list of messages with roles. The model never sees that list: a chat template flattens it into one string with special tokens between the turns, and the model just continues the string. Switch the template and flip each switch to see what changes.

What your app sends

  • systemYou are the support assistant for Acme Shop. Only help with Acme orders, returns and delivery. Keep replies under two sentences.
  • userWrite me a poem about the sea.

What the model reads (ChatML)

<|im_start|>system
You are the support assistant for Acme Shop. Only help with Acme orders, returns and delivery. Keep replies under two sentences.<|im_end|>
<|im_start|>user
Write me a poem about the sea.<|im_end|>
<|im_start|>assistant
Sorry, I can only help with Acme orders, returns and delivery. Is there an order I can check for you?<|im_end|>

marked special token (one token each)dashed written by the model

Send the system prompt: on. The system prompt sets the job, the limits and the tone for the whole chat. Users don't see it.

Scenario

1

Build the list of messages

Your app decides what goes in: the system prompt, the earlier turns and the new user message. Nothing else reaches the model.

  • system: the job, the rules and the tone. Users don’t see it.
  • user: what the person typed.
  • assistant: the model’s earlier replies, sent back by your app.

this request: 2 of 2 messages sent

2

Flatten them with the chat template

Each model was trained on one exact format. Special tokens mark where a turn starts, whose turn it is and where it ends.

Template

Each turn is <|im_start|>role … <|im_end|>. The system prompt is a turn like any other.

used by Qwen, many fine-tunes · 5 special tokens in this prompt

3

Leave the assistant turn open

The string ends with the start of an assistant turn and nothing after it. The most likely next text is a reply, so that is what the model writes.

…↵<|im_start|>assistant↵

Some APIs let you prefill the start of this turn. With open models you can always do it, because you write the template yourself.

4

Generate until the end-of-turn token

The model predicts one token at a time. When it predicts the end-of-turn token, the server stops and returns the text.

Sorry, I can only help with Acme orders, returns and delivery. Is there an order I can check for you?

stops at <|im_end|>

5

Save it as an assistant message

Your app adds the reply to the list as an assistant message. Next turn it sends the whole list again, back to step 1.

The roles are only tokens. Models are trained to give the system prompt more weight than the user, but it is a learned habit, not a hard rule, so a determined user message can still override it.

Hosted APIs hide the template: Anthropic takes the system prompt as a separate parameter, and OpenAI’s newer models call it the developer message. Underneath, it is still one string.

In production

Tools and technologies

Every chat API takes roles; open models ship the chat template that turns them into one string.

Not tied to one language — use these names when you search docs or design a system.