Introducing EvalCI: A Self-Hosted CI Gate for LLM Quality

A few weeks ago we wrote about why we run LLM evals in CI like tests, not as a one-off notebook — a fixed question set, automated grading, and a merge-blocking threshold, the same discipline unit tests have applied to regular code for two decades. EvalCI is the tool we built to make that the default setup instead of something every team has to assemble themselves.

The gap between "we run evals" and "evals gate the merge"

Plenty of teams run evals. Most compare a new run against a rolling "recent average" that drifts every time a dataset or config changes, and most surface the result as a dashboard someone might check — not a pass/fail signal a CI pipeline can act on before merge. EvalCI is built around a narrower, stricter idea: an explicit baseline you set deliberately, and a severity-aware exit code that actually fails the build when a change regresses past a threshold you configure.

How it works

1. Set an explicit baseline — the last run you actually reviewed and approved, referenced by ID. Every future comparison is against that specific run, never an inferred average.
2. Run and score on every change — LLM-as-judge and lexical metrics, triggered by a push, a schedule, or a manual call.
3. Gate the merge — a regression at or above your configured severity fails the check; a warning-level dip doesn't block a critical-gated pipeline. You choose the bar per pipeline.

Self-hosted from day one, not from an enterprise sales call

If eval traffic can't leave your infrastructure — and for most teams working with real or client-derived data, it shouldn't — "contact sales" isn't an onboarding flow. EvalCI runs in your own cluster from the first `docker compose up` for local development, and on Kubernetes for production, with the API tier autoscaling under an HPA and scheduling running as a dedicated single-replica deployment.

The detection engine is open, on purpose

evalci-core — the regression detector, trend analysis, and scoring engine — is Apache-2.0 and installable on its own, with the real test suite (48 tests, ported directly from the production suite, none mocked away) available alongside it. `pip install evalci-core` gets you a zero-infrastructure demo: no Docker, no database, no API key required to see the detection logic run against injected regressions.

How this compares to the observability platforms

Braintrust, LangSmith, and Langfuse are the names that come up first for LLM evaluation, and all three are strong, broad tools — tracing, agent monitoring, and eval scoring together in one platform. EvalCI is deliberately narrower: it isn't trying to be a full observability suite (that's on the roadmap, not shipped today), it's built specifically around an explicit, pinned baseline and severity-aware CI exit codes as the core primitive, which none of the three treats as central the way a merge-gate needs. Self-hosting matters here too — it's available from day one via Docker Compose or Kubernetes, not gated behind an enterprise tier the way it typically is with Braintrust or LangSmith.

Where we're at

We're onboarding a small number of design partners directly rather than opening self-serve signup yet — teams shipping LLM features weekly, engineering-led, who want self-hosted infrastructure or data control. Every design partner works directly with us on setup and has real input into what gets built next. If that's useful for what you're building, the EvalCI page has the details on applying, or you can go straight to evalci-core on GitHub if you just want to see the engine.

For the concrete setup — pinning a baseline, wiring the CI gate, choosing a severity threshold — see how to add LLM regression testing to a CI/CD pipeline. If you're deciding between this and a broader open-source testing framework, see the evalci-core vs Promptfoo comparison, and for a full production deployment on the observability side, see the ForgeObserver case study.

Frequently asked questions

Is EvalCI free to use?

evalci-core, the open-source detection engine, is Apache-2.0 and free to install and run on its own. EvalCI itself, the hosted/self-hosted platform, is currently onboarding design partners directly rather than running self-serve pricing — see the EvalCI page for how to apply.

Does EvalCI require sending our data to a third party?

No — it's self-hosted from day one via Docker Compose for local development or Kubernetes for production, so eval traffic stays in your own infrastructure rather than needing an enterprise contract to get that guarantee.

How is EvalCI different from just running evals in a notebook?

A notebook run tells you the system looked good once. EvalCI runs your golden dataset automatically on every relevant change, compares against an explicit baseline you pinned deliberately, and fails the CI check with a severity-aware exit code when a regression crosses your configured threshold — turning "did this get worse" into an automated gate instead of a manual, easy-to-skip check.

Christian Chukwuka

Founder & AI Systems Engineer