Skip to content
Optimal Lab

AI Engineering · interactive

Chain of thought and reasoning models

A model does the same amount of work for every token. Ask for the answer straight away and every intermediate result has to happen in one pass. Let it write the steps first and each result becomes text it can read. See when that helps, when voting over several chains helps more, and what it costs.

Question · 4 steps

A café sells 23 coffees an hour from 7:00 to 11:00, then 15 an hour until 15:00. Each coffee is $4. How much does it take in?

Model output · Think step by step

  1. 7:00 to 11:00 is 4 hours: 4 × 23 = 92 coffees.
  2. 11:00 to 15:00 is 4 hours: 4 × 15 = 60 coffees.
  3. 92 + 60 = 152 coffees.
  4. 152 × $4 = $608.
  5. Answer: $608.
0%50%100%123456steps in the problem
direct 23%step by step 86%vote of 5 98%

Direct 23%, step by step 86%, a vote of 5 98%. The longer the problem, the wider the gap.

Question

How it answers

1

Ask a question

Count the steps a person would write down to solve it. A fact is one; a word problem can be many.

A café sells 23 coffees an hour from 7:00 to 11:00, then 15 an hour until 15:00. Each coffee is $4. How much does it take in?

4 steps · answer $608

2

Answer directly

The first answer token is computed in one pass through the layers. Every intermediate number has to be worked out inside that pass, with nowhere to write it.

“$368.” ✗

0.97 × 0.62^3 = 23% right · 1 token

3

Write the steps out

“Think step by step” makes the model write each result down. The next token can read it, so every step is a small, easy one.

  1. 7:00 to 11:00 is 4 hours: 4 × 23 = 92 coffees.
  2. 11:00 to 15:00 is 4 hours: 4 × 15 = 60 coffees.
  3. 92 + 60 = 152 coffees.
  4. 152 × $4 = $608.
  5. Answer: $608.

0.97 × 0.97^4 = 86% right · 37 tokens

4

Sample several chains and vote

Self-consistency: sample a few chains at a temperature above zero and keep the most common final answer. A slip in one chain rarely repeats in the others.

Chains

  • $608: ●●● 3
  • $628: ● 1
  • $548: ● 1

a majority right: 98% · 185 tokens

5

Pay for the thinking

Reasoning tokens are output tokens: billed, and slower to produce than reading the prompt. Spend them where a problem has steps.

directstepsvoteA fact97%94%100%Two steps60%91%99%Four steps23%86%98%Six steps9%81%95%

Reasoning models (OpenAI o-series, Claude with extended thinking, DeepSeek-R1) are trained with reinforcement learning to do this on their own: they think in hidden tokens before answering, and you set a budget for how many.

The per-step rates are a teaching model. The shape (direct falls fast, written steps fall slowly, votes lift it) follows Wei et al. 2022 and Wang et al. 2022.

In production

Tools and technologies

Prompted chain of thought works with any model; reasoning models do it on their own, with a budget you set.

Not tied to one language — use these names when you search docs or design a system.