The LLM Cost Curve Nobody Models: Retry Storms

We've written before about attributing LLM cost accurately per customer and per feature. That model — cost per call times calls per customer — is correct as far as it goes, and it misses the single most common cause of an unexplained cost spike: retries. Specifically, retries that aren't bounded, backed off, or circuit-broken, quietly multiplying the cost of every failed call by however many times your code was willing to try again.

How this actually happens

A model provider has a rough ten minutes — elevated latency, a wave of 500s, rate limiting kicking in harder than usual. Nothing dramatic, the kind of thing that happens periodically with any external dependency. Code with naive retry logic — retry immediately, retry up to some large number of times, or worse, no retry limit at all — responds to that ten minutes by hammering the provider harder, which is exactly the wrong direction: more load on an already-struggling service, and every one of those retries is a billed call, succeeding or not, if the provider bills for the attempt rather than strictly for a successful response.

The multi-step chain problem is worse

The version of this that actually blows up a bill isn't a single retried call — it's a multi-step agent chain that retries the whole chain from the beginning when any single step fails. A five-call chain where step four fails and the retry logic reruns steps one through four instead of just step four turns one failure into four redundant, billed calls before the fifth even gets attempted again. This is the pattern worth specifically auditing for, because it's the one that's easy to write by accident — "just retry the request" is the simplest thing to implement, and for a single call it's also correct.

The fix, in order of impact

1. Retry the failed step only, never the whole chain — this alone usually has the biggest effect on runaway multi-step cost.
2. Exponential backoff with jitter, not immediate or fixed-interval retry — spreads retry load out instead of synchronizing a thundering herd against a provider that's already struggling.
3. A hard ceiling on retry attempts per logical call, after which it fails loudly instead of trying indefinitely.
4. A circuit breaker that stops calling a provider entirely for a cooldown window once its recent failure rate crosses a threshold, instead of continuing to retry (and pay for) calls to a service that's clearly degraded.

What to actually monitor

Cost-per-customer dashboards catch steady-state overspend. They're usually too coarse-grained to catch a retry storm quickly, because it shows up as a spike inside a single day that a daily or weekly rollup smooths over. The metric that catches it fast is retry rate as its own tracked number — not inferred from a cost spike after the fact, but alerted on directly the moment it crosses a threshold, the same way you'd alert on error rate for any other service dependency.

Retries need idempotency to be safe, not just cheap

There's a second, sharper failure mode hiding inside naive retries: if the retried step has a side effect — charging a customer, sending a notification, calling an external API that isn't safe to call twice — a retry after an ambiguous failure (the call may have succeeded server-side even though the client never got a response) can execute that side effect more than once. Bounding and backing off retries controls cost; making the underlying action idempotent controls correctness, and a system that only fixes the cost side can still double-charge a customer during exactly the kind of degraded-provider window that triggered the retries in the first place.

We've written before about attributing LLM cost accurately per customer and per feature — that model tells you where cost is going; this one explains the multiplier that breaks it. For the idempotency side specifically, see idempotency keys for AI agents that take actions.

Frequently asked questions

How many times should an LLM call be retried?

There's no universal number, but an unbounded or very high retry ceiling is the actual danger — a hard cap (commonly in the low single digits per logical call) combined with exponential backoff and a circuit breaker for sustained provider degradation is safer than tuning the exact count.

Does exponential backoff alone solve the cost problem?

Backoff spreads retries out over time, which helps avoid synchronizing a thundering herd against a struggling provider, but it doesn't cap total cost on its own — a call that eventually succeeds after many backed-off attempts still gets billed for each attempt. A hard retry ceiling and a circuit breaker are what actually bound the cost, not backoff by itself.

Christian Chukwuka

Founder & AI Systems Engineer