Errors, rate limits & retries
Model APIs fail routinely - overloaded servers, rate limits, timeouts - and production code must expect, absorb and recover from all of it.
In plain terms
A model API is a wildly popular restaurant: sometimes there's a queue (429 rate limit), sometimes the kitchen is slammed (529/500), sometimes your order times out. An amateur app crashes; a professional app waits politely, retries, and has a plan B.
Why it matters
This is the #1 gap between demo and production. Your demo works because you called the API 10 times. Production calls it 100,000 times, and at that volume every rare failure happens daily.
How it works
Key statuses: 429 too many requests (you exceeded requests- or tokens-per-minute), 500/529 provider trouble, 400 your request is malformed (retrying won't help!), timeouts. Standard remedy: exponential backoff with jitter - wait 1s, 2s, 4s, 8s (+random) between retries, max ~5 attempts, only for retryable errors. SDKs have this built in; configure, don't reinvent.
When you use it
In the wrapper around every model call, from the first day. Also at design time: rate limits shape architecture (queues, batching) for any high-volume feature.
Common mistakes
- Retrying
400errors forever - a malformed request never fixes itself. - Retrying instantly in a tight loop, which makes the rate-limiting worse (thundering herd).
- No timeout, so one hung request freezes a user forever.
- No fallback plan when the provider has a bad hour.
Best practices
- Set explicit timeouts on every call.
- Log every failure with request id - providers can trace them.
- For volume jobs, respect the rate-limit headers and use batch APIs.
- Consider a fallback model/provider for availability-critical paths.
Try it yourself
Force each failure on purpose: send an invalid model name (400), fire 100 concurrent requests (429), set a 1-ms timeout. Make your wrapper survive all three with clean log lines.
Project: Phase project - terminal assistant. A CLI chatbot with: streaming replies, persistent conversation memory saved to a JSON file, one working tool (calculator or web search), a system prompt with a real personality, retry-with-backoff error handling, and per-session token/cost reporting. This one project exercises every node in Phases 00-02.
Resources
- Anthropic - Errors & rate limits Status codes and what to do about each.