API design for AI services: idempotency, retries, timeouts, and error contracts
'It works' isn't an answer; 'two retries with backoff, then a 502 with a typed error' is. How to design an API that behaves predictably when the model — or anything upstream — fails.
An AI endpoint fails more often, and less predictably, than a CRUD one: the model times out, rate-limits, or returns junk. So the senior question isn't 'does it work?' — it's 'what does your service do when it doesn't?'. A good answer is a designed contract, not a shrug.
Idempotency: make retries safe
Clients retry and networks duplicate, so a request can arrive twice. If a retried 'create itinerary' produces two itineraries, that's a bug you built. An idempotency key fixes it: the client sends a unique key per logical operation, the server records the key with its result, and any repeat of that key returns the original result instead of doing the work again. Stripe's API is the reference design. Key on the operation, store the outcome, dedupe the repeats.
Retries, timeouts, and backoff
Put a timeout on every call to a model or dependency — never wait forever. Retry a bounded number of times, with exponential backoff and jitter, and only on transient failures (timeouts, 429s, 503s); don't retry a deterministic 400, because it will just fail again. 'Two retries with backoff, then give up' is a real, defensible policy you can say out loud. 'It retries until it works' is not a policy — it's an outage waiting to happen. And circuit-break a dependency that's clearly down rather than hammering it (the distributed-systems post).
Error contracts: what you return when it breaks
This is the part that separates senior from mid: define what your API returns on failure as a contract. A consistent error shape (a machine-readable code, a human message, and a request_id to trace it), the right status for the cause (502/503 when an upstream model is unavailable, 429 when you're rate-limiting, 504 on timeout, 422 on bad input), and enough information for the client to decide whether retrying is worth it. 'The model failed, so we return 502 with {code: 'upstream_unavailable', request_id}' is the answer. Silence, or a 500 with an HTML stack trace, is the tell that failure was never designed.
Streaming and partial failure
For long model calls, stream the response (SSE) so the client sees progress and perceived latency drops (the streaming post). But a stream can die halfway, so the contract has to say what a truncated response looks like and how the client knows it was incomplete — a terminal 'done' event, or an error frame. Designing the happy stream is easy; designing the interrupted one is the job.
Why the contract is the product
Clients integrate against your failures as much as your successes. A predictable, documented error contract is what lets them build reliably on top of you; an API that fails a different way every time is one no one can trust. Give the failure modes the same care as the happy path — for an API, the way it breaks is part of what you're shipping.
The measure of an API isn't how it behaves when everything works — it's whether 'the model timed out' produces the same, predictable, retryable response every single time.