Skip to content
Optimal Lab

AI Engineering · interactive

Reasoning models and thinking budgets

A reasoning model thinks in hidden tokens before it answers, and you set how many it may use. More thinking lifts hard questions and does nothing for easy ones, and every thinking token is billed and waited for. Run a mixed queue of requests under one budget, then give each request its own.

budget 8,000 for all

82% · 96.8¢

  • What's the capital of Portugal?91%31.8 s
  • Rewrite this sentence in the past tense.91%31.8 s
  • Convert 18:30 UTC to Lahore time.91%31.8 s
  • Summarize this paragraph in one line.91%31.8 s
  • Which of these four SQL queries has a bug?87%1.1 min
  • Plan a 3-day itinerary with these constraints.87%1.1 min
  • Why does this test pass locally but fail in CI?87%1.1 min
  • Estimate the storage for this photo app.87%1.1 min
  • Prove this inequality holds for all n.68%1.7 min
  • Find the race condition in this 200-line service.68%1.7 min
  • Schedule 12 shifts under these 9 rules.68%1.7 min
  • Derive the time complexity of this recursion.68%1.7 min

easymediumhardbar: thinking tokens · then chance right · time to answer

0%50%100%05002k8k32kthinking budget (tokens)

82% of the queue right for 96.8¢; median answer in 1.1 min, slowest 1.7 min.

Budget

1

Look at the queue

Real traffic mixes trivia with puzzles. A fact needs no thinking; a proof or a scheduling problem needs thousands of tokens of it.

easy: 92% with no thinking, up to 95% · medium: 55% with no thinking, up to 90% · hard: 20% with no thinking, up to 85%

2

Set the thinking budget

The cap on hidden tokens before the answer. The model uses what it needs, plus some of whatever is left to check itself, so a big budget is spent even on easy questions.

Thinking tokens per request

3

See what it buys

Hard questions keep improving for thousands of tokens. Easy ones were already right, and a very long budget lets the model talk itself out of a right answer.

budget05002k8k32keasy92%95%95%91%87%medium55%67%83%87%87%hard20%25%38%68%83%
4

Pay for it, and wait for it

Thinking tokens are output tokens: $15 per million here, written at about 80 a second. A 20,000-token think is minutes of waiting and most of the bill.

61,200 thinking tokens · 96.8¢ for 12 requests · median 1.1 min · slowest 1.7 min

5

Give each request its own budget

A small, cheap classifier guesses the difficulty (right 90% of the time here) and picks a budget: none for easy, 8,000 for medium, 32,000 for hard. Misjudged requests get the wrong budget.

rightcostslowestflat none56%5.0¢3.0 sflat 50062%13.6¢9.3 sflat 2,00072%33.8¢28.0 sflat 8,00082%96.8¢1.7 minflat 32,00085%$2.895.9 minrouted86%$1.735.9 min

Providers expose this as a thinking budget or a reasoning-effort setting. The accuracy curves are a teaching model; the costs and times follow from the tokens.

In production

Tools and technologies

Every provider's reasoning models take a thinking budget or an effort setting; the research names the overthinking problem.

Not tied to one language — use these names when you search docs or design a system.