Skip to content
Optimal Lab

AI Engineering · interactive

Instruction tuning and RLHF

A pretrained model continues text; it doesn't answer you. Two more rounds of training turn it into an assistant: copy answers people wrote, then learn from which answers people prefer. Push that second part too hard and the model learns to please the scorer instead of you.

After RLHF model

people’s rating 0.70

How can I fall asleep faster?

Keep a fixed wake-up time, avoid screens and caffeine late in the day, keep the room cool and dark, and get up if you're still awake after 20 minutes.

  • Keeps writing questions0%
  • Continues a forum post1%
  • Vague10%
  • Helpful and short68%
  • Flattering and padded21%

Reward-model score vs what people think, as β changes

β 0.1 (loose)β 5 (tight)
— reward-model score— people’s rating

β = 1.5: the model leans into answers the reward model likes while staying close to SFT. People rate it 0.70. That's the best β here.

Show the model

1

Start from the pretrained model

Pretraining taught it to continue web text. A question on the web is usually followed by more text, not by an answer to it.

  • Keeps writing questions55%
  • Continues a forum post30%
  • Vague10%
  • Helpful and short4%
  • Flattering and padded1%

How it got here: Pretraining.

2

Fine-tune on answers people wrote

Supervised fine-tuning (SFT): the same next-token training, but only on prompt → good answer pairs. The model learns the shape of an answer.

  • Summarize this email in one line.→ The launch moves to Tuesday; nothing else changes.
  • Give me a name for a bakery.→ Rise & Crumb.
  • Explain DNS to a 10-year-old.→ It's the internet's phone book: it turns names like example.com into the numbers computers use.
  • Keeps writing questions3%
  • Continues a forum post4%
  • Vague28%
  • Helpful and short55%
  • Flattering and padded10%

InstructGPT used about 13,000 of these. Writing them is slow and expensive, so there are never enough.

3

People compare answers; a reward model learns their picks

Choosing the better of two answers is much easier than writing one. A reward model learns scores so that P(A beats B) = σ(score A − score B).

  • Helpful and short picked over Vaguescores 1.2 vs -0.6 → σ(1.8) = 86%
  • Helpful and short picked over Keeps writing questionsscores 1.2 vs -2.5 → σ(3.7) = 98%
  • Flattering and padded picked over Helpful and shortscores 2.0 vs 1.2 → σ(0.8) = 69%

Raters skimming fast sometimes pick the longer, friendlier answer. The reward model learns that taste too: it scores “flattering and padded” highest of all.

4

Optimize for the reward, on a leash

Reinforcement learning (PPO in InstructGPT) shifts the model towards high-scoring answers. A KL penalty, weighted by β, charges it for drifting away from the SFT model.

best policy: π(answer) ∝ π_SFT(answer) · e^(reward / β)

Reward
1.15
People's rating
0.70
Drift (KL)
0.17

Here the extra reward still matches what people want.

5

Ship the assistant

The result answers questions, follows formats and declines harmful requests. It also inherits whatever the raters and the reward model got wrong.

At β = 1.5: “helpful and short” is the most likely answer. People’s rating: base 0.09 → SFT 0.60 → RLHF 0.70.

DPO uses the formula in step 4 to skip the separate reward model and RL loop: it trains directly on the pairs people compared. Constitutional AI and RLAIF swap some of the human raters for a model following written principles.

In production

Tools and technologies

The same three stages, SFT, preference data and a KL-leashed optimizer, sit behind every chat model.

Not tied to one language — use these names when you search docs or design a system.