Setting Up Golden Datasets for LLM Regression Testing

Most teams that set up LLM evals start with the model and the scoring method — which grader, what threshold, LLM-as-judge or exact-match. The dataset underneath all of it gets far less attention, and it's usually the thing quietly determining whether the whole eval suite is actually useful. A regression suite run against ten easy, obvious examples will pass almost anything; the value is entirely in what the dataset actually covers.

Fixed and versioned, not resampled

The dataset needs to be the same set of examples every time a regression check runs — not a fresh sample pulled from somewhere each time. If the questions change between runs, a score difference could mean the model changed, or it could just mean the dataset changed, and there's no way to tell which. Version the dataset the same way code gets versioned, and change it deliberately, not implicitly.

Structure by domain or scenario, not one flat list

A single undifferentiated list of test prompts makes regressions hard to localize — a score drop tells you something got worse, not what. Splitting the dataset by domain or scenario turns a vague "score dropped 4%" into an actionable "the payments-support domain dropped 12%, everything else is flat," which is a completely different debugging problem. When building the internal eval suite used for evalci-core's own CI, the dataset is split across four distinct domains — fintech and payments support, rental and property search, a WhatsApp business assistant, and general safety scenarios — specifically so a regression in one domain doesn't get diluted into an aggregate score.

Size for coverage, not a round number

There's no universal "you need N examples" rule — the actual requirement is coverage of the failure modes that matter for the system in question. A narrow, well-defined task might be meaningfully covered by a few dozen carefully chosen examples. A broader assistant handling varied user intent needs enough spread across those intents that a regression in one specific area doesn't get lost in an average across everything else. More examples in a category that never actually fails doesn't add coverage — it adds noise and run time.

Include the hard cases on purpose

It's tempting to build a dataset out of clean, representative examples — the kind of input the system is expected to see most of the time. That's necessary but not sufficient. The examples that actually catch regressions are the edge cases: ambiguous phrasing, adversarial or malformed input, known-hard questions from a specific domain, and past real failures that got fixed once and should never silently come back. A dataset without deliberately-included hard cases will pass a genuinely worse model, because nothing in it was hard enough to catch the regression.

Mix deterministic checks with LLM-as-judge

Not every example needs the same grading method. Where there's a clear right answer — a specific field extracted correctly, a schema that must validate, a fact that's either right or wrong — deterministic checks (exact-match, regex, schema validation) are faster, cheaper, and don't drift. For open-ended quality — tone, faithfulness to a source document, whether a response actually resolved the user's intent — LLM-as-judge grading is usually necessary, with its own fixed rubric so judging itself doesn't become an unversioned, drifting variable in the suite.

Once a dataset like this exists, running it automatically on every change — rather than manually, occasionally — is the other half of the picture. See how to wire LLM regression testing into a CI/CD pipeline for the mechanics of that step.

Frequently asked questions

How many examples should a golden dataset have?

There's no fixed number — it depends on how many distinct scenarios or domains the system needs to cover well. A narrow task can be meaningfully covered by a few dozen carefully chosen examples; a broad assistant needs enough spread across intents that a regression in any one area is still visible.

Should a golden dataset include failure cases?

Yes — deliberately. Past real failures that got fixed, known-hard or ambiguous inputs, and adversarial edge cases are what actually catch a regression. A dataset built only from clean, easy examples will pass a genuinely worse model.

The takeaway

The eval framework and grading method matter, but the dataset underneath determines whether any of it catches a real regression. Fixed and versioned, structured by domain so a regression can be localized, sized for coverage rather than a round number, and deliberately including hard cases — that's what turns a golden dataset from a formality into the thing that actually keeps a bad change from shipping.

Christian Chukwuka

Founder & AI Systems Engineer