Phase 03 · Talking to models: the API layer Core

Streaming responses

Receiving the answer token-by-token as it's generated, instead of waiting for the whole thing - the difference between a live app and a frozen one.

In plain terms

Without streaming, asking for a long answer is like ordering coffee and staring at a blank counter for 30 seconds. With streaming, you watch the cup fill. Same coffee, same wait - but one feels instant and the other feels broken.

Why it matters

Generation is genuinely slow - a long answer takes 10-30 seconds. Users abandon apps that sit silent for 5. Streaming converts unavoidable latency into perceived speed, which is why every serious chat UI streams.

How it works

Models generate one token at a time anyway. With stream: true, the server pushes each chunk over Server-Sent Events (SSE) the moment it exists; your code loops over an iterator, printing/forwarding chunks, then assembles the full text at the end. To reach a browser, your backend re-streams those chunks to the frontend.

When you use it

Any answer a human watches being written. Skip it for machine-to-machine calls (extraction, classification) where only the final result matters.

In code

with client.messages.stream(
    model="claude-sonnet-5", max_tokens=800,
    messages=[{"role":"user","content":"Explain tides to a child"}]
) as stream:
    for chunk in stream.text_stream:
        print(chunk, end="", flush=True)

Common mistakes

Building the whole app non-streaming, then discovering retrofitting streams touches every layer - decide early.
Parsing JSON out of a half-finished stream (it's invalid until complete).
Forgetting error handling mid-stream - connections die at chunk 47 too.

Best practices

Stream anything user-facing; buffer machine-facing calls.
Show the stream immediately but also log the assembled final response for debugging and evals.

Try it yourself

Take your terminal chatbot and switch it to streaming - print chunks with end="", flush=True so words appear as they're born. Feel the UX difference at 30 lines of code.

Resources

Anthropic - Streaming docs Event types and code for real apps.