Skip to content
Optimal Lab

AI Engineering · interactive

Speech to text and text to speech

A voice agent chains three models: speech to text, an LLM, and text to speech. Every stage adds delay, and the first stage can mishear names and numbers. Talk to it in a quiet room and a busy café, and get the reply time from four seconds down to about one.

You

speak

End of turn

600 ms silence

Speech to text

40% WER

LLM

1.8 s

Text to speech

700 ms

until you hear a reply

3.6 s

  • waiting for silence 600 ms
  • speech to text 500 ms
  • LLM 1.8 s
  • text to speech 700 ms
  • ┊ a person: ~200 ms

restart the pig bouncer pool on staging and page maria if check out errors past two percent

Marked: misheard or extra words. Struck through: words it missed.

3.6 s until you hear a reply, and the agent heard “Restart the pig bouncer pool on staging and page Maria if check out errors past two percent”.

You say: “Restart the PgBouncer pool on staging and page Priya if checkout errors pass two percent”

Step 1 of 5

  1. 1

    Detect the end of your turn

    A voice-activity detector waits for a stretch of silence. Too short and it talks over you when you pause to think; too long and every reply feels slow.

    Silence before replying

    cuts you off on ~6% of turns

  2. 2

    Transcribe it

    Speech-to-text turns audio into words. Noise, names and product terms cause most errors. Word error rate counts substitutions, missing and extra words against what was said.

    Where you are

    WER = 6 / 15 = 40% · 500 ms after you stop

  3. 3

    Think of a reply

    The LLM reads the transcript. If you wait for its whole answer before speaking, the user waits too. Stream it: hand each finished sentence to the voice as soon as it's written.

    whole reply in 1.8 s

  4. 4

    Speak it

    Text-to-speech turns the reply into audio. A streaming voice starts playing the first sentence while the rest is still being made.

    700 ms to the first sound

  5. 5

    Add it up

    What the user feels is the gap between finishing a sentence and hearing a reply. People in conversation usually leave about 200 ms. Speech-to-speech models skip the text hops altogether.

    600 ms + 500 ms + 1.8 s + 700 ms = 3.6 s

    Stage times are typical figures, not one vendor’s. The ~200 ms human gap is from Stivers et al. (2009), across ten languages.

In production

Tools and technologies

Voice agents combine speech-to-text, an LLM and text-to-speech, or use one speech-to-speech model.

Not tied to one language — use these names when you search docs or design a system.