Catch the regression before it merges, not after a customer files a ticket.
EvalCI runs your evals on every change, compares the result against a baseline you set explicitly, and fails the build when quality drops — self-hosted, in your own infrastructure, from day one.
Every eval tool tells you the score changed. Almost none of them block the merge.
Code gets a test suite and a CI gate. Prompt and model changes usually get someone eyeballing five outputs before shipping — until the regression that slips through becomes a support ticket.
No explicit baseline
Most eval platforms compare against "recent runs," which drifts every time your dataset or config changes. You want to know: did it get worse than the version you actually approved.
No real CI gate
A dashboard that shows a dropped score after the fact isn't a gate. A severity-aware exit code that fails the check before merge is.
Self-hosting locked behind Enterprise
If your eval traffic can't leave your infrastructure, "contact sales" isn't an onboarding flow. EvalCI runs in your cluster from the first `docker compose up`.
An Honest Comparison, Not a Hero Chart
The eval-tooling space is real and getting crowded. Here’s where EvalCI is actually different — and where it isn’t.
| EvalCI | Braintrust | LangSmith | Langfuse | |
|---|---|---|---|---|
| Self-hosted from day one | Yes | Enterprise only | Enterprise only | Yes |
| Explicit regression baseline | Core primitive | Score-based | Score-based | Score-based |
| Severity-aware CI exit codes | Yes | Via Action | Via integration | Manual |
| Open-source core engine | Apache-2.0 | Closed | Closed | MIT |
| Full observability suite (tracing, agents) | Roadmap | Yes | Yes | Yes |
Three Steps. The Third One Is the One That Matters.
This is a real sequence — each step depends on the one before it — so it’s numbered because the order actually carries information.
Set an explicit baseline
Pick the last run you actually approved. Every future comparison is against that run — never an inferred "recent average" that drifts.
baseline_run_id = "run_8f2a1c"Run and score on every change
LLM-as-judge and lexical metrics run against your dataset. Triggered by a push, a schedule, or a manual call.
evals run --wait --git-branch=$BRANCHGate the merge
A regression at or above your configured severity fails the check. A warning-level dip doesn't block a critical-gated pipeline — you choose the bar.
--fail-on-regression criticalRuns Where Your Data Already Lives
Local dev on Docker Compose. Production on your own Kubernetes cluster. Nothing about your eval traffic has to leave your network.
Local, in one command
API, worker, scheduler, dashboard, Postgres, Redis, object storage — one stack, one command.
# clone, set your model API key, go
docker compose up -d
make migrateProduction, autoscaled
API tier scales under an HPA. Scheduling runs as a dedicated single-replica deployment — never duplicated across replicas.
# apply the manifests
kubectl apply -f infra/kubernetes/The Detection Engine Is Yours to Read, Run, and Audit
Not a watered-down teaser — the actual regression detector, trend analysis, and scoring engine, with the real test suite ported alongside it.
- Zero-infra demo — no Docker, no database, no API key
- 48 tests ported directly from the production suite
- Apache-2.0, importable as a real dependency
See the full CI/CD setup walkthrough, or how evalci-core compares to Promptfoo.
We’re Onboarding a Small Number of Teams Directly
No self-serve signup yet — on purpose. Every design partner gets the founder on setup, direct input into what gets built next, and free access while we build together.
EvalCI is a product of ChuksForge AI Solutions Ltd · RC No. 9523804 · Live at evalci.chuksforge.com