Why We Run LLM Evals in CI Like Tests, Not as a One-Off Notebook

Most teams building with LLMs have run an eval at some point — a notebook, a spreadsheet of test prompts, a manual pass through twenty examples before a demo. Almost none of them run that same eval automatically on every prompt change, model swap, or retrieval update. That gap is where regressions live.

The notebook trap

A notebook run once tells you the system was good at one point in time, against one version of the prompt. It says nothing about whether the change someone is about to merge makes things better or worse, because nobody re-runs the notebook on every change — it's manual, slow, and easy to skip under deadline pressure. This is exactly the failure mode unit tests were invented to prevent for regular code, and it applies just as directly here.

What an eval suite needs to look like a test suite

• A fixed, versioned set of test questions or scenarios — not ad-hoc examples someone thinks of that morning.
• An automated grader for each response — LLM-as-judge, exact-match, or a rules-based check, depending on the task.
• A pass/fail threshold that blocks a merge or deploy, not just a dashboard someone might look at later.
• Execution on every relevant change — prompt edits, model version bumps, retrieval strategy changes — via CI, the same trigger as unit tests.

LLM-as-judge grading: useful, with caveats

Using a strong model to grade another model's output works well for open-ended quality checks — faithfulness, completeness, tone — where exact-match scoring doesn't apply. The caveat is that the judge model needs its own fixed rubric and, ideally, periodic calibration against human-graded examples, or grading drift becomes its own silent regression.

Regression detection is the actual point

The value isn't the score on any single run — it's the diff between runs. A retrieval strategy change that drops accuracy on a specific category of question, a model swap that quietly increases hallucination rate, a system-prompt edit that improves one metric while degrading another: none of these show up from eyeballing a few examples, but they show up immediately in a scored, versioned eval run compared against the previous one.

What this catches that manual review doesn't

Manual review catches the failure someone happens to test for. An automated, comprehensive question set catches the failure nobody thought to check — the edge case in a category the reviewer wasn't focused on that day. At scale, that's the difference between a regression caught in CI and one reported by a customer.

The takeaway

Treat evals as a CI gate, not a pre-demo ritual. A fixed question set, automated grading, and a merge-blocking threshold turn "does this still work" from a hopeful guess into a number checked on every change — which is the entire point of having tests in the first place.

Christian Chukwuka

Founder & AI Systems Engineer