Skip to content
Optimal Lab

AI Engineering · interactive

Small and on-device models

A small model on the user's own device is free to run, private, and has no network delay. It is also worse at anything that needs real reasoning, and it can't read a long document. Pick a model for each job a feature does and see quality, speed and cost add up.

taskOn-device 3BSmall hostedFrontier hostedTag support ticketsPull fields from invoicesSummarize an emailWrite a SQL queryPlan a multi-step tripAnswer from a 100-page contract

Click a cell to pick that model for the task. Scores are a teaching model of a typical pattern.

average quality
92%
per month
$19,583
stay on device
0 / 6

Frontier for everything: 92% average quality for $19,583 a month.

Pick for me

1

List the jobs

A “smart inbox” feature is really several jobs: tag, extract, summarize, query, plan, read long files. Each can use a different model.

  • Tag support tickets 150 in · 5 out
  • Pull fields from invoices 800 in · 60 out
  • Summarize an email 600 in · 80 out
  • Write a SQL query 300 in · 60 out
  • Plan a multi-step trip 400 in · 300 out
  • Answer from a 100-page contract 60,000 in · 100 out
2

Score each model on each job

Run your own test set per job. Small models hold up on narrow, repetitive work and fall apart on multi-step reasoning. Fine-tuning a small model on one job narrows the gap on that job only.

Tagging 82% → 93%, invoices 78% → 94%, trip planning 25% → 30%.

3

Check the speed

On-device: no network, but a phone writes about 20 tokens a second. Hosted: a few hundred ms of network and queue, then fast. Short answers favour the device; long ones the server.

devicesmallfrontierTag support tickets625 ms363 ms633 msPull fields from invoices5.0 s860 ms1.8 sSummarize an email5.5 s953 ms2.0 sWrite a SQL query3.8 s760 ms1.6 sPlan a multi-step trip16.0 s2.4 s5.6 sAnswer from a 100-page contract—13.0 s22.2 s
4

Price it

Hosted models charge per token, in and out. On-device costs you nothing per request: the user's hardware does the work.

Requests per job, per month

per request: device $0.00/1k · small $0.09/1k · frontier $3.00/1k (summaries)

5

Pick per job

Use the smallest model that's good enough for each job, keep private data on the device where you can, and send the hard jobs up.

  • Tag support ticketsFrontier hosted · 94%
  • Pull fields from invoicesFrontier hosted · 95%
  • Summarize an emailFrontier hosted · 93%
  • Write a SQL queryFrontier hosted · 90%
  • Plan a multi-step tripFrontier hosted · 88%
  • Answer from a 100-page contractFrontier hosted · 90%

The 100-page contract alone is most of the frontier bill: 60,000 input tokens a request.

In production

Tools and technologies

Small open models run on phones and laptops; the platforms ship runtimes and their own on-device models.

Not tied to one language — use these names when you search docs or design a system.