How to Add LLM Regression Testing to Your CI/CD Pipeline

Running evals in CI conceptually is one thing; actually wiring a merge-blocking gate is another. This is the concrete setup with EvalCI — the same self-hosted, Apache-2.0-core tool this site runs its own regression checks with.

Step 1: Set an explicit baseline

Every comparison needs a fixed point of reference, and it should be a run someone actually reviewed and approved — not an inferred "recent average" that quietly drifts as new runs get folded in. Pin the baseline to a specific run ID:

baseline_run_id = "run_8f2a1c"

Every future comparison runs against exactly that run, until a new one is deliberately approved to replace it. This is the core primitive EvalCI is actually built around — most eval tools compare against a rolling or inferred score, which makes "did this get worse" a fuzzier question than it needs to be.

Step 2: Run and score on every change

With a golden dataset in place, trigger a run on every push, on a schedule, or via a manual call — LLM-as-judge and lexical metrics score the output against the dataset automatically:

evals run --wait --git-branch=$BRANCH

This is the same trigger model as a unit test suite: it runs on the change, not on a schedule someone remembers to check manually.

Step 3: Gate the merge by severity

Not every score dip should block a deploy — a warning-level regression on a minor edge case shouldn't stop a critical hotfix from shipping, but a severe drop absolutely should. Severity-aware exit codes let a pipeline set that bar explicitly rather than treating every regression as equally blocking:

evals run --wait --git-branch=$BRANCH --fail-on-regression critical

A regression at or above the configured severity fails the check and blocks the merge. Anything below that threshold surfaces in the report without stopping the pipeline — the bar is configurable per pipeline, not a fixed all-or-nothing gate.

Why self-hosted matters here specifically

Eval traffic often includes real or representative user input — the same data governance concerns that apply to production traffic generally apply to what gets sent through an eval run. EvalCI runs in your own cluster from the first `docker compose up`, with the detection engine (evalci-core) Apache-2.0 and pip-installable, rather than self-hosting being an enterprise-tier feature gated behind a sales conversation.

pip install evalci-core
python examples/regression_demo.py

For teams evaluating whether to build this on evalci-core directly versus another open-source option, see the evalci-core vs Promptfoo comparison.

Frequently asked questions

Can this block a merge, not just report a score?

Yes — that's the specific design goal. A severity-aware exit code (`--fail-on-regression critical`, or a lower threshold) fails the CI check itself, the same way a failing unit test would, rather than just surfacing a dashboard someone has to remember to check.

Does this require sending eval data to a third-party service?

No — EvalCI is self-hosted from day one via Docker Compose or Kubernetes, and the core detection engine (evalci-core) is Apache-2.0 and runs in your own infrastructure.

The takeaway

An explicit baseline, a CI-triggered run against a golden dataset, and a severity-aware exit code are what turn "we should really check this before shipping" into an actual gate — the same guarantee a unit test suite gives regular code, applied to prompt and model changes instead of just application logic.

Christian Chukwuka

Founder & AI Systems Engineer