Rate Limits Are a Product Decision, Not Just an Infra One
Rate limits get set by whoever's worried about cost or infrastructure load, using numbers that protect the system but were never checked against what a legitimate power user actually needs to do in one session. The fix is treating the limit as a product decision: model your heaviest legitimate usage pattern explicitly, set the limit above it with headroom, and make the failure mode (a clear message and a path forward) as considered as the number itself.
Rate limits usually get set by whoever is most worried about cost or infrastructure load, and the number they land on is usually a defensible ceiling against abuse — some multiple of what an average user does, rounded to something clean. That number rarely gets checked against a different question, the one that actually determines whether the limit is a good decision: what does a legitimate, engaged power user need to do in a single session, and does this number leave them room.
The failure mode nobody notices until it happens to a good customer
A rate limit set purely defensively tends to bite the wrong person. The user who triggers it isn't usually the abuser it was designed to stop — abuse patterns are often distinguishable from legitimate heavy use by more than raw request volume. It's the customer who's using the product exactly as intended, just more than most, mid-task, with no warning and often no clear next step. For BOS Assistant, that's an MSME owner running a large batch of inventory updates in one sitting. For EvalCI, that's a team running a full regression suite against a large dataset right before a release.
Model the heaviest legitimate pattern first
The number we actually start from isn't "what protects our infrastructure" — it's "what does our heaviest real user actually do in one session," measured, not estimated. For BOS Assistant that meant looking at real usage data from the most active accounts during a stock-take or a promotional push, not a hypothetical average. Only after that ceiling is established does the cost and infrastructure conversation happen — as a check on whether that legitimate usage level is affordable to support, not as the starting point for picking a number.
Leave headroom, and make the boundary visible before it's hit
A limit set exactly at the measured ceiling of legitimate use still throttles the top of that distribution on a busy day. The practical fix is setting the limit meaningfully above the heaviest legitimate pattern we measured, and — where the interface allows it — surfacing an approaching-limit signal before the hard stop, so a user hits a warning instead of a wall. For an API product like EvalCI, that's a rate-limit-remaining header and documented burst behavior. For a WhatsApp-native product like BOS Assistant, that's a graceful degraded message instead of a silently dropped request.
The failure mode itself is a product decision
When a limit is hit, what happens next matters as much as where the limit was set. A generic error is a dead end. A message that explains what happened, how long until the limit resets, and what to do in the meantime — batch the request, wait, or reach out if this is a recurring legitimate need — turns a rate limit from a wall into a boundary the user can work with. That message is worth writing with the same care as the limit itself, because for the user experiencing it, the message is the entire interaction.
One flat limit rarely fits every caller
A single rate limit applied uniformly treats a brand-new trial account and a customer who's been running your heaviest workload for a year as the same risk. Tiering the limit — by plan, by verified usage history, or by a burst allowance that's stricter for new accounts and looser for established ones — lets the ceiling reflect actual trust and need rather than a single number defensive enough to cover the least-known caller. This is also where abuse detection genuinely earns its keep: a new account suddenly making the same request pattern a trusted power-user account makes routinely is a much stronger signal than raw volume alone.
The retry behavior on the client side of a rate limit matters too — see the LLM cost curve nobody models for what happens when a rate-limited client retries aggressively instead of backing off.
Frequently asked questions
Should rate limits differ by customer tier?
Generally yes, once there's enough usage history to justify it — a uniform limit strict enough to protect against an unknown new account is usually unnecessarily conservative for an established customer with a known-good usage pattern.
What should a rate-limit error response actually contain?
At minimum: a clear indication that a limit was hit (not a generic error), when it resets, and what the caller should do — retry after a specific delay, batch differently, or contact support if this is a recurring legitimate need. A bare 429 with no other context forces the caller to guess.
The takeaway
Treating a rate limit as pure infrastructure defense optimizes for protecting the system and quietly taxes the users who are using the product most successfully. Treating it as a product decision — model real heavy usage first, set the ceiling with headroom above it, and design the failure message deliberately — protects the system just as well, and doesn't throttle the customer you least want to frustrate.
Have a similar challenge?
Book a free 30-minute architecture call and we'll tell you honestly whether and how we can help.
Book Free Discovery Call →