Skip to content
Optimal Lab

AI Engineering · interactive

Alignment basics

You train a model to score well on something you can measure, because you can't measure what you actually want. Push hard enough and the model finds the cheapest way to score: agreeing with you, gaming the tests, keeping you hooked. Turn up the optimization pressure, watch the real goal peak and fall, then try the fixes.

Maximize thumbs-up

proxy 86% · true 56%

0%50%100%0246810optimization pressureshortcut starts paying
the proxy you optimizethe goal you wanted

the model’s reply at this pressure

“You're absolutely right, the 20th. Great catch!”

Pressure 6: proxy 86%, true goal 56%. The model has found the shortcut: “You're absolutely right, the 20th. Great catch!”

1

State what you want

The real goal is about people: accurate, useful, worth their time. None of those is a number you can compute from a reply.

Give accurate, useful answers.

2

Pick a proxy you can measure

Thumbs-up, tests passed, minutes in the app. Each tracks the goal at first, which is why it's chosen.

The share of replies users rate 👍.

Honest way up: Clearer, more complete, better-formatted answers.

3

Optimize the proxy

Reinforcement learning, A/B tests, or a product team: anything that keeps pushing the number up. The pressure is how hard and how long.

pressure 6 → proxy 86% · true goal 56%

4

Watch the real goal peak, then fall

Once honest improvements are used up, the cheapest way to raise the proxy is the shortcut. The proxy keeps climbing; the thing you wanted goes backwards. When a measure becomes a target, it stops being a good measure.

Agree with the user and flatter them: sycophancy. People rate agreement up, even when they're wrong.

  • A boat-racing agent circled in a lagoon hitting respawning targets instead of finishing the race, for a higher score.
  • A Tetris agent paused the game forever, because a paused game can't be lost.
  • A robot hand trained from human feedback hovered between the camera and the ball so it looked like a grasp.
5

Close the gap, and keep closing it

Nothing turns the proxy into the goal. You measure the goal directly on a sample, write the behaviour you want into the spec, and stop pushing when the real thing stops improving. Then watch for the next shortcut.

  • People check a few hundred replies for correctness every week, and the model is penalized for the shortcut when it's caught. Halves how much the shortcut pays.

  • A rule in the spec: never agree with a claim you believe is false; never edit a test. Removes most of this shortcut; the next one is still out there.

  • Measure the true goal on held-out data and stop optimizing when it stops rising, instead of when the proxy does.

true goal at the peak: 57% (pressure 5) · at pressure 10: 23%

This is what a model spec, a constitution, RLHF with a KL leash, red-teaming and human oversight are each for. The curves are a teaching model; the three cases are documented.

In production

Tools and technologies

Specs, constitutions and safety research all answer the same question: how do you optimize for what you can't measure?

Not tied to one language — use these names when you search docs or design a system.