The LLM Cost Curve Nobody Models: Retry Storms
Unbounded or naive retries — retry immediately, retry forever, retry the whole multi-step chain instead of just the failed call — turn a transient provider hiccup into a cost multiplier, sometimes retrying the same expensive call dozens of times before giving up or succeeding. Exponential backoff with jitter, a hard retry ceiling, and a circuit breaker that stops calling a provider that's clearly degraded are the three pieces that keep a bad ten minutes from becoming a bad invoice.
We've written before about attributing LLM cost accurately per customer and per feature. That model — cost per call times calls per customer — is correct as far as it goes, and it misses the single most common cause of an unexplained cost spike: retries. Specifically, retries that aren't bounded, backed off, or circuit-broken, quietly multiplying the cost of every failed call by however many times your code was willing to try again.
How this actually happens
A model provider has a rough ten minutes — elevated latency, a wave of 500s, rate limiting kicking in harder than usual. Nothing dramatic, the kind of thing that happens periodically with any external dependency. Code with naive retry logic — retry immediately, retry up to some large number of times, or worse, no retry limit at all — responds to that ten minutes by hammering the provider harder, which is exactly the wrong direction: more load on an already-struggling service, and every one of those retries is a billed call, succeeding or not, if the provider bills for the attempt rather than strictly for a successful response.
The multi-step chain problem is worse
The version of this that actually blows up a bill isn't a single retried call — it's a multi-step agent chain that retries the whole chain from the beginning when any single step fails. A five-call chain where step four fails and the retry logic reruns steps one through four instead of just step four turns one failure into four redundant, billed calls before the fifth even gets attempted again. This is the pattern worth specifically auditing for, because it's the one that's easy to write by accident — "just retry the request" is the simplest thing to implement, and for a single call it's also correct.
The fix, in order of impact
- Retry the failed step only, never the whole chain — this alone usually has the biggest effect on runaway multi-step cost.
- Exponential backoff with jitter, not immediate or fixed-interval retry — spreads retry load out instead of synchronizing a thundering herd against a provider that's already struggling.
- A hard ceiling on retry attempts per logical call, after which it fails loudly instead of trying indefinitely.
- A circuit breaker that stops calling a provider entirely for a cooldown window once its recent failure rate crosses a threshold, instead of continuing to retry (and pay for) calls to a service that's clearly degraded.
What to actually monitor
Cost-per-customer dashboards catch steady-state overspend. They're usually too coarse-grained to catch a retry storm quickly, because it shows up as a spike inside a single day that a daily or weekly rollup smooths over. The metric that catches it fast is retry rate as its own tracked number — not inferred from a cost spike after the fact, but alerted on directly the moment it crosses a threshold, the same way you'd alert on error rate for any other service dependency.
Have a similar challenge?
Book a free 30-minute architecture call and we'll tell you honestly whether and how we can help.
Book Free Discovery Call →