Introducing EvalCI: A Self-Hosted CI Gate for LLM Quality

A few weeks ago we wrote about why we run LLM evals in CI like tests, not as a one-off notebook — a fixed question set, automated grading, and a merge-blocking threshold, the same discipline unit tests have applied to regular code for two decades. EvalCI is the tool we built to make that the default setup instead of something every team has to assemble themselves.

The gap between "we run evals" and "evals gate the merge"

Plenty of teams run evals. Most compare a new run against a rolling "recent average" that drifts every time a dataset or config changes, and most surface the result as a dashboard someone might check — not a pass/fail signal a CI pipeline can act on before merge. EvalCI is built around a narrower, stricter idea: an explicit baseline you set deliberately, and a severity-aware exit code that actually fails the build when a change regresses past a threshold you configure.

How it works

1. Set an explicit baseline — the last run you actually reviewed and approved, referenced by ID. Every future comparison is against that specific run, never an inferred average.
2. Run and score on every change — LLM-as-judge and lexical metrics, triggered by a push, a schedule, or a manual call.
3. Gate the merge — a regression at or above your configured severity fails the check; a warning-level dip doesn't block a critical-gated pipeline. You choose the bar per pipeline.

Self-hosted from day one, not from an enterprise sales call

If eval traffic can't leave your infrastructure — and for most teams working with real or client-derived data, it shouldn't — "contact sales" isn't an onboarding flow. EvalCI runs in your own cluster from the first `docker compose up` for local development, and on Kubernetes for production, with the API tier autoscaling under an HPA and scheduling running as a dedicated single-replica deployment.

The detection engine is open, on purpose

evalci-core — the regression detector, trend analysis, and scoring engine — is Apache-2.0 and installable on its own, with the real test suite (48 tests, ported directly from the production suite, none mocked away) available alongside it. `pip install evalci-core` gets you a zero-infrastructure demo: no Docker, no database, no API key required to see the detection logic run against injected regressions.

Where we're at

We're onboarding a small number of design partners directly rather than opening self-serve signup yet — teams shipping LLM features weekly, engineering-led, who want self-hosted infrastructure or data control. Every design partner works directly with us on setup and has real input into what gets built next. If that's useful for what you're building, the EvalCI page has the details on applying, or you can go straight to evalci-core on GitHub if you just want to see the engine.

Christian Chukwuka

Founder & AI Systems Engineer