Streaming LLM responses: why the tokens arrive one at a time
That typewriter effect isn't a gimmick — it's the model streaming tokens as it generates them. Here's how streaming works, why it transforms perceived speed, and where it gets tricky.
Ask a chatbot a question and the answer appears word by word. That's not an animation for flavour — it's the model sending each token the instant it's generated, instead of making you wait for the whole response. Streaming is one of the highest-leverage UX decisions in an LLM app, and it's worth understanding on both sides of the wire.
Why stream at all
A long answer might take ten seconds to finish generating. Without streaming, the user stares at a spinner for all ten. With streaming, the first words show up in a few hundred milliseconds and the rest flow in as they're produced. The total time is the same — but the perceived latency, the thing users actually feel, collapses.
How it works
The provider sends the response as a stream of small chunks (typically Server-Sent Events), each carrying the next token or two. Your code reads the stream, appends each delta as it arrives, and renders progressively. Turning it on is usually a single flag; consuming it is a loop.
const stream = await client.chat.completions.create({
model: 'gpt-4o',
messages,
stream: true,
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content ?? '';
process.stdout.write(delta); // render as it arrives
}The trade-offs to plan for
- Partial JSON — if you asked for structured output, you can't parse mid-stream; buffer until it's complete, or stream to the UI but validate at the end.
- Error handling — a stream can fail halfway, so design for a response that stops partway through.
- Post-processing — guardrails and moderation that need the whole answer have to run after the stream closes, not during.
Streaming doesn't make the model faster. It makes the wait disappear — and for users, that's the same thing.