Why Your LLM Eval Scores Don't Match Production Behavior
A passing eval suite and degrading production behavior usually diverge for one of three reasons: the golden dataset has gone stale relative to what real users are actually asking now; the LLM-as-judge grading model has drifted without anyone re-calibrating it against human judgment; or an aggregate score is hiding a real regression in one specific slice of traffic that got averaged out by everything else performing fine. None of these are solved by a better single score — they're solved by treating the dataset, the judge, and the aggregation method as things that need their own periodic maintenance, not one-time setup.
This is one of the more disorienting moments in running an LLM product: the eval suite passes, sometimes at a genuinely high score, and production complaints are climbing at the same time. It feels like a contradiction. It usually isn't — it's a sign that the eval suite is measuring something slightly different from what's actually happening in production, in one of a few predictable ways.
Cause 1: the dataset has gone stale
A golden dataset built at launch reflects what users were asking at launch. Real usage drifts — new use cases emerge, phrasing patterns shift, an entirely new category of question starts showing up that nobody anticipated when the dataset was built. An eval suite that's never updated to reflect that drift will keep testing the same, increasingly outdated slice of the input space, and can stay green indefinitely while genuinely failing on the part of production traffic it was never built to cover.
The fix is treating the golden dataset as a living artifact — periodically sampling real production failures (with appropriate privacy handling) and folding representative, anonymized versions of them back into the dataset, so it evolves alongside actual usage instead of staying frozen at whatever it looked like on day one.
Cause 2: the judge model has drifted
LLM-as-judge grading is only as reliable as the judge itself, and judges aren't static. A judge model gets updated by its provider, a prompt tweak to the grading rubric changes what "good" means without anyone noticing, or the judge simply has systematic blind spots that never got caught because it was never checked against actual human judgment in the first place. A score of 92% from a judge that's quietly become more lenient over time isn't a stable measurement — it's a number that means something different than it did three months ago.
Periodic calibration — spot-checking a sample of judge-graded outputs against a human rating, on a recurring cadence rather than once at setup — is the only real defense here. Without it, judge drift is invisible by construction: the score looks consistent because the judge's own standard moved along with whatever it's grading.
Cause 3: aggregation is hiding a real regression
A single overall pass rate can look completely healthy while one specific slice of it has genuinely degraded — a particular domain, a specific user intent, a certain input length or language. If the eval suite reports one blended number across everything, a real regression in a smaller but important slice gets mathematically diluted by everything else performing normally, and the aggregate simply never moves enough to trip an alert.
This is the direct argument for scoring and reporting by segment rather than only in aggregate — the same domain-split structure that makes a golden dataset useful for localizing regressions in the first place also makes production-vs-eval mismatches easier to actually diagnose instead of just noticing they exist.
Frequently asked questions
How often should a golden dataset be updated?
There's no universal cadence, but treating it as a one-time setup task is the mistake to avoid — periodically sampling real, representative production traffic (anonymized appropriately) and folding it back in keeps the dataset aligned with how usage actually evolves.
Does LLM-as-judge grading need to be checked against human judgment?
Yes, on a recurring basis rather than only at initial setup. Judge models and grading rubrics drift over time, and without periodic calibration against human ratings, that drift is invisible — the score stays consistent-looking even as what it actually measures shifts.
The takeaway
An eval suite that passes while production quality genuinely degrades isn't a contradiction — it's usually a sign that the dataset, the judge, or the aggregation method has drifted out of sync with reality. All three need periodic maintenance, not just initial setup, for the score to keep meaning what it's supposed to mean.
Have a similar challenge?
Book a free 30-minute architecture call and we'll tell you honestly whether and how we can help.
Book Free Discovery Call →