Voice agent latency: where the silence comes from

A voice agent's response time is the pause between a caller finishing a sentence and the agent starting to speak. It is the sum of four stages: detecting that the caller has finished, speech recognition, the language model's time to first token, and the first audio from text-to-speech. Three of those are fairly steady. The language model is the one that occasionally takes seconds, and that is what callers hear as dead air.

The stages of one turn

Stage	What happens	What makes it slow
End of turn	The agent decides the caller has stopped speaking	Waiting too long to be sure; background noise
Speech recognition	The caller's audio becomes text	Waiting for a final transcript instead of using a streaming one
Language model	The model reads the conversation and starts its reply	Busy providers, long prompts, reasoning before the answer
Text-to-speech	The first words become audio	Waiting for a whole sentence; a slow voice model
Network and telephony	Audio and requests travel between all of the above	Distance between the caller, your agent and each service

The model's part is its time to first token, not the time to finish the reply: text-to-speech starts on the first words and the rest streams in faster than it can be spoken.

Why the language model is the unpredictable stage

Speech recognition and text-to-speech take about the same time on every turn. A large language model does not. Most turns start quickly; a few take a second, two seconds, or never start before your timeout. Nothing fails loudly enough to raise an error. The request is simply late.

A call has many turns, so a small share of slow ones reaches most calls. If 5% of requests stall, a call of 20 turns has about a 64% chance of at least one. The arithmetic is in why first tokens stall.

What a slow turn costs

• The caller talks over the agent. After a second of silence people say "Hello?" or repeat themselves, and the agent then has two overlapping things to respond to.
• The agent falls back. If your agent times out on the first token and hands the turn to a smaller model, the caller gets a weaker answer in the middle of the call.
• Teams give up on the smart model. The common fix is to move to a small, fast model for every turn, which answers on time and handles tools and long conversations worse.

How to reduce voice agent latency

1. Stream everything. Use streaming speech recognition, stream the model's reply, and start text-to-speech at the first clause instead of the full response.
2. Keep the prompt short, and keep its start the same. The model reads the whole prompt before it answers. Put instructions and tools first and the changing conversation last, so the repeated part can be cached from one turn to the next.
3. Send one session ID per call. With FasterTokens, set X-Session-Id to the call's ID so later turns can reuse the cached prompt.
4. Turn reasoning off for spoken turns. Thinking tokens are silence. FasterTokens serves DeepSeek V4.1 Flash 1017 with reasoning off.
5. Cap the reply. A spoken reply is a sentence or two. A low max_tokens keeps a turn from running long.
6. Set a first-token deadline and keep a fallback model. Decide how long a turn may wait, and fail over cleanly when it is exceeded. FasterTokens ends a turn with HTTP 504 if no output is produced within 2 seconds (you can change the deadline per request), before any part of a response is sent, so a fallback can take the turn without the caller hearing a restart. See voice agent setup.
7. Do not depend on one provider's worst moment. Every inference provider has slow moments. For every request FasterTokens queries multiple inference providers simultaneously and optimizes for time to first token, so one of them stalling rarely slows the turn.
8. Cover tool calls with speech. When a turn has to wait on a tool, have the agent say so ("one moment while I check") instead of going quiet.
9. Measure the tail, around the clock. Log time to first token for every turn and watch the 95th and 99th percentiles by hour of day. Averages hide the turns callers remember.

Do you have to choose between smart and fast?

The trade most teams make is a small model that is quick and a large one that is capable. The large model's median is usually fast enough for a call; its tail is what rules it out. Take the tail away and the smart model becomes usable for voice.

That is what FasterTokens is for: an OpenAI-compatible endpoint that serves DeepSeek V4.1 Flash 1017 for voice agents, with its time to first token measured in public. You change a base URL and a key. Start with the quickstart.