Why LLM APIs stall on the first token

A first-token stall is a request to a language model that takes a second or more to begin answering when it usually takes a fraction of that. It happens on every inference provider, it raises no error, and averages hide it. For a voice agent it is the difference between a reply and dead air.

What a stall looks like

The same request, sent to the same model on the same provider, starts in 400 milliseconds most of the time and in 1,500 or 3,000 milliseconds once in a while. Sometimes it does not start before your timeout. Nothing is reported as failed: the response arrives, late.

Why it happens

Shared capacity. A large model is expensive to serve, so providers typically run it on hardware many customers share. When a burst of requests arrives, yours waits its turn before the model starts on it.
New prompts. A prompt the provider has not seen recently has to be read in full; one it has seen can be cached. The first turn of a call, or a turn that lands on different hardware, pays the full price.
Long prompts. Reading time grows with the prompt. Instructions, tool definitions and a long call history all sit in front of the first token.
Busy hours. Load follows the working day. A provider that is fast at night can be slow in the afternoon.

None of this is a bug to be fixed once. It is how shared inference behaves, which is why it has to be measured continuously.

P50, P95 and P99

Latency is reported in percentiles because the average says little about the turns that go wrong:

Figure	Meaning
P50 (median)	Half of requests were at least this fast
P95	19 of every 20 requests were at least this fast; 1 in 20 was slower
P99	99 of every 100 requests were at least this fast; 1 in 100 was slower

A provider can have an excellent P50 and a P99 measured in seconds. The gap between the two is the tail, and the tail is what a caller remembers.

A small stall rate reaches most calls

A phone call is many requests in a row, so the chance that a call has at least one slow turn is much higher than the chance for any single request. For a call of 20 turns:

Share of requests that stall	Chance a 20-turn call has at least one
1%	18%
2%	33%
5%	64%
10%	88%

These rows are arithmetic, not measurements: the chance is 1 − (1 − rate)^20. The measured stall rate for a single provider serving DeepSeek V4.1 Flash 1017, and for the same requests through FasterTokens, is on the benchmark page.

Why a timeout and a retry are not enough

A timeout turns a stall into a failure you can act on, which is better than waiting. But the caller has already sat through the timeout, and then waits again for the retry or the fallback. If the fallback is a smaller model, the answer is weaker too. A timeout limits how bad a stall can get; it does not make stalls rare.

What makes stalls rare

Different providers are rarely slow at the same moment. An agent that depends on one provider gets that provider's tail. FasterTokens queries multiple inference providers simultaneously on every request and optimizes for time to first token, so a stall on one of them rarely reaches the caller. If a turn still gets no first token in time, it fails fast with a clear error so your fallback can take it.

How to measure your own tail

Record every turn. Log time to first token per request, with the hour and the kind of turn.
Report percentiles. P50, P95 and P99, and the share of turns over a threshold that matters to you, such as one second.
Count the ones that never answered. A turn that timed out is the slowest turn, not a missing one. Dropping it flatters the numbers.
Measure all day. An hour of testing at a quiet time tells you nothing about the busy hours.

The FasterTokens benchmark is run this way: typical voice turns, evenly spaced around the clock, with unanswered turns counted as the slowest, and a single provider measured on the same requests for comparison.